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Abstract — Any unconstrained information inequality in three 
or fewer random variables can be written as a linear combination 
of instances of Shannon's inequality I(A;B\C) > 0. Such 
inequalities are sometimes referred to as "Shannon" inequalities. 
In 1998, Zhang and Yeung gave the first example of a "non- 
Shannon" information inequality in four variables. Their tech- 
nique was to add two auxiliary variables with special properties 
and then apply Shannon inequalities to the enlarged list. Here 
we will show that the Zhang- Yeung inequality can actually be 
derived from just one auxiliary variable. Then we use their same 
basic technique of adding auxiliary variables to give many other 
non-Shannon inequalities in four variables. Our list includes 
the inequalities found by Xu, Wang, and Sun, but it is by no 
means exhaustive. Furthermore, some of the inequalities obtained 
may be superseded by stronger inequalities that have yet to 
be found. Indeed, we show that the Zhang- Yeung inequality is 
one of those that is superseded. We also present several infinite 
families of inequalities. This list includes some, but not all of the 
infinite families found by Matus. Then we will give a description 
of what additional information these inequalities tell us about 
entropy space. This will include a conjecture on the maximum 
possible failure of Ingleton's inequality. Finally, we will present 
an application of non-Shannon inequalities to network coding. 
We will demonstrate how these inequalities are useful in finding 
bounds on the information that can flow through a particular 
network called the Vamos network. 



I. Introduction 

For collections A, B, and C of jointly related discrete 
random variables, denote the entropy of A by 

H(A) :=5^-p(o)tog 2 (p(a)), 



where 01og 2 (0) := 0. The conditional entropy of A given B 
is defined by 

H{A\B) := H{AB) — H(A) (1) 
= J2 -P((M))log 2 (p((a,fc))) 

(a.b)eAB 
a£A 

the mutual information between random variables A and B 

by 

I(A; B) := H(A) + H(B) - H(AB), 

and the conditional mutual information between random vari- 
ables A and B given C by 

I(A; B\C) := H{AC) + H(BC) - H(C) - H(ABC). 

The basic inequalities H(A) > 0, H(A\B) > 0, and 
I (A; B) > were originally proved in 1948 by Shannon J5] 
and can all be obtained as special cases (e.g. see ifTTl ) of the 
inequality 



I{A;B\C) > 0. 



(2) 



'Key words: Entropy, Information Inequalities. 
2 Mathematical Reviews: 26A12. 



For example, letting C be a trivial random variable with just 
one element, we obtain 

I(A;B\C) = H(AC) + H(BC) - H(C) - H(ABC) 
= H(A) + H{B) - - H(AB) 
= I(A;B) 

yielding that I (A; B) > 0. Similarly, H(A) = I(A; A) > 
and H(A\B) = I (A; A\B) > 0. 

By combining together instances of these basic inequalities, 
other valid inequalities can be formed. For example, 



I(A;B\C) + 2I(A;C) > 



(3) 
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holds for any random variables A, B, C, since each of the 
terms separately is at least zero. Applying the definitions, this 
can be rewritten as 

< H(AC) + H(BC) — H(ABC) — H(C) (4) 
+2H(A) + 2H(C) - 2H{AC). 

Canceling and rearranging the terms gives us 

< H(BC)-H(ABC)+H(C) (5) 
+2H(A) - H(AC). 

If we now permute the variables A^B, B^C, C^A 
then this becomes 

< H(AC) - H(ABC) + H(A) (6) 
+2H(B) - H(AB). 

Throughout our discussion, we will identify inequalities that 
can be derived from each other using definitions, basic alge- 
braic manipulation, and rearrangement of the variables. Thus, 
the inequalities ([3]), ©, ©, © are really just different forms 
of the same inequality. 

Since all conditional entropies and all conditional mutual 
informations can be written as linear combinations of joint 
entropies, we give the following definition. 

Definition 1: Let n be a positive integer, and let Si, ... , Sk 
be subsets of {1, ...,n}. Let aj £ R for 1 < i < k. An 
inequality of the form 

aiH({Ai : i e Si}) + ■ ■ ■ + akHdA, : i e S k }) > 

is called an information inequality if it holds for all jointly 
distributed random variables Ai, . . . A n . 
The textbook ifTTI refers to information inequalities as "the 
laws of information theory". As an example, taking p = 2, 
Si = {1}, S 2 = {2}, S 3 = 0, S 4 = {1, 2}, ai = a 2 = 1, and 
a 4 = -1, one obtains H(A X ) + H(A 2 ) - H(A 1 ,A 2 ) > 0, 
which is an information inequality since it is always true (this 
can be more succinctly expressed as I{A\\ A 2 ) > 0). 

Information inequalities that can be derived by adding 
special cases of Shannon's original inequality will be given 
a special designation bearing his name. 

Definition 2: A Shannon information inequality is any in- 
formation inequality of the form 

^aJiAiiBilCi) >0 (7) 

i 

where each a, > 0. 

Any information inequality that cannot be expressed in the 
form (0 will be called a non-Shannon information inequality. 
It is known ITT1 p. 308] that all information inequalities 
containing three or fewer random variables are Shannon in- 
equalities. 

These were the only known types of information inequalities 
until Zhang and Yeung in 1998 published a "non-Shannon" 
information inequality |[T5l IfTTI Theorem 14.7 on p. 310]. 

Theorem 1 (Zhang-Yeung Theorem): The following is a 4- 
variable non-Shannon information inequality: 

2I(C; D) < I(A; B)+I{A; C, D)+3I(C; D\A)+I(C; D\B). 



There are two parts to Theorem Q] The first part claims 
that the inequality is valid, and the other part claims that 
it is non-Shannon. To prove this inequality is valid, Zhang 
and Yeung added two auxiliary variables and then applied 
Shannon inequalities to the enlarged list of variables. This 
technique of adding auxiliary variables will be encapsulated 
by the Copy Lemma |2]given in Section HTlbelow. We will then 
give a new proof of the first part of the Zhang-Yeung theorem 
in Section [TTIJ Unlike the original proof, this proof requires 
only one auxiliary variable. The second part of Theorem Q] 
can be proved using the Information Theory Inequality Prover 
(ITIP)(see (12)), which is a MATLAB program for veri- 
fying testing whether an inequality is a Shannon inequality. It 
was written and made freely available by Yeung and Yan. 

Since the seminal work fl5l . many other non-Shannon 
iformation inequalities have been found. See for exam- 
ple, Lnenicka Q. Makarychev, Makarychev, Romashchenko, 
and Vereshchagin J6j, Zhang JT3), Zhang and Yeung lfl4l . 
Dougherty, Freiling, and Zeger |]2], and Matus JE). 

In this paper we will give the results of a systematic search 
for additional four-variable non-Shannon inequalities using the 
same basic technique of Zhang and Yeung, as given in the 
Copy Lemma. In Section |IV] with a will present the general 
methodology that was used for this search. The depth of the 
search is measured by how many auxiliary variables were 
used and how many instances of the Copy Lemma were used. 
Ignoring inequalities that can be derived from others by a 
permutation of the variables, it turns out that the Zhang-Yeung 
inequality is the only one that can be derived from just one 
auxiliary variable. 

In Section [V] we exhaustively search for all inequalities that 
can be derived from just two auxiliary variables. The result of 
this search (after weeding out redundant inequalities that can 
be deduced from the others) is the list of six two-copy-variable 
inequalities that appear in J2j. 

As the list of inequalities grows, it turns out that some of the 
earlier inequalities are no longer needed. To see if an inequality 
is still necessary, we temporarily remove it from the list and 
perform a linear program to see if the inequality in question 
can fail when all of the others are satisfied. If not, then it has 
been "superseded" by the others and is trimmed from the list 
permanently. The Zhang-Yeung inequality is an example of 
one that has been superseded. Sharing the fate of the Zhang- 
Yeung inequality, each of the six inequalities from Section [V] 
has also been superseded. 

Next, we exhaustively searched for all inequalities that can 
be derived using three auxiliary variables. These are split into 
two sections. In Section [VTl we present those that only require 
two instances of the Copy Lemma, and in Section rVHl we give 
those that require three instances of the Copy Lemma. In both 
of these sections, we list only the inequalities that have not 
yet been superseded. We also give a computer generated proof 
of each one. To save space, the first few proofs are given in 
detail while the latter proofs are abbreviated. 

At this point, we were reaching the limit of what could be 
feasibly done with the resources available. To speed things up, 
the use of the ITIP program was eventually replaced with a 
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faster C-program. We began searching for inequalities using 
four variables and only three instances of the Copy Lemma. 
This search eventually finished; it took an estimated 50-100 
CPU-years to complete. We were not able to complete the 
entire exhaust over four auxiliary variables. We estimate that 
this search would take about 70 times longer. Our final list 
of inequalities appears in Section IVIIII To save space, rather 
than giving the proof of each one, we provide the sequence 
of copy steps used in the proof. 

The first two infinite families of non-Shannon inequalities 
were found by Matus (8). Each of these families is given in 
list form, indexed by the positive integers. The first list was 
used by Matus to prove the fundamental fact that no finite 
collection of linear inequalities will ever be able to describe the 
entropy space completely. In Section HXl we also look at entire 
classes of inequalities. These will be presented as rules, which 
allow us to automatically generate new inequalities from old 
ones. These rules can also be iterated leading to uncountable 
collections of information inequalities. These rules can also 
be used to generate many individual lists of inequalities. As 
an example, we will show how to derive the first list of 
Matus from these rules. Matus' second list, however, was not 
uncovered by this process. 

Additional information inequalities and a third infinite list 
have been discovered by Xu- Wang-Sun ifTOl . In Section IXl we 
will show how to derive stronger versions of their inequalities 
from the list in Section IVIII and from the inequality rules of 
Section HH 

In Section [XI] we will summarize what we have learned 
about the structure of entropy space. We will give some 
volume computations showing that, in fact, these inequalities 
do not seem to be closing the gap between the space satisfying 
the Shannon inequalities and the space of known entropic 
vectors. We present a certain probability distribution with 
just four atoms, that we believe gives the maximum possible 
failure of Ingleton's inequality. We conclude that some new 
techniques will likely be necessary in order to settle this "four- 
atom" conjecture. 

The Zhang-Yeung inequality has recently been applied to 
network coding to demonstrate that Shannon information in- 
equalities are in general insufficient for computing the coding 
capacity of a network (see and JT]). In Section IXIII we 
will apply our list of inequalities to improve the information 
theoretic upper bound for the coding capacity of a simple 
network called the Vamos network. 



II. Copy Lemmas 

There is only one known basic technique for coming up 
with new information inequalities. 

1) Start with a set of arbitrary random variables. 

2) Add auxiliary random variables with special properties. 

3) Apply known information inequalities to the enlarged 
set of random variables. 

In this section, we will present the methods for obtaining the 
auxiliary variables. These are encapsulated in the following 
lemma, which is essentially due to Zhang and Yeung. 



Lemma 2 (Copy Lemma): Let A, B, C, D be jointly dis- 
tributed random variables. Then there is another random vari- 
able R, jointly distributed with A,B,C,D with the following 
properties. 

CI. The marginal distributions of (A,B,C) and 

(A,B,R) are the same with R replacing C. 
C2. I(CD; R\AB) = 

In this case we say that R is a Z?-copy of C over (A, B). 

Proof: Let A, B, C, D, denote the alphabets of the ran- 
dom variables A, B, C, D resp. Let a, b, c, d denote arbitrary 
elements of A, B, C, D, resp. with probability p(a, b, c, d). Let 
R be a new random variable and let r denote an arbitrary 
element of its alphabet, which is C. Define the joint probability 
distribution of A, B, C, D, R by 



p'(a,b,c, d,r) 



p{a,b, c,d)Y ld P( a > b , r, d) 



It is clear that these are nonnegative. Summing over r we get 

p(a,b, c,d) J2 r dP( a >b,r, d ) 



^2p'(a,b,c, d, r) 



p(a,b, c,d) 



so that p' is an extension of p, which also implies that the 
sum of all of the probabilities p' is 1 . Similarly, the marginal 
distribution of (A, B, R) is given by 

v— v / , , , , J2 c .dP( a ' b ' c ' d )JldP( a , b , r , d ) 
> p (a,b,c,d,r) = ■ = —r — T 

= ^2p{a,b,r,d) 

d 

while the marginal distribution of (A, B, C) is given by 
2~2dP{a,b,c,d), demonstrating (CI). 

If we write (C2) in terms of entropies, we get 
H(ABCD) + H(ABR) - H(AB) - H(ABCDR) = 0. But 
H{A,B,R) = H(A 1 B 1 C) by (CI), so it remains to show 
that H(ABCDR) = H(ABCD) + H(ABC) - H(AB). We 
compute H(ABCDR) as 



^ -p'(a, b, c, d, r) log 2 (p'(a, b, c, d, r)) 

a,b,c,d,r 

^2 -p'( a Ac,d,r)log 2 (p(a,&,c,eO) 



.b.c.d.r 



^2 -p'(a, b, c, d, r) log 2 (^ p(a, b, r, d)) 

a,b,c,d,r 



c.d 



But 



X] -p'( a ,b,c,d,r)\og 2 {p(a,b,c,d)) 

X] -p( a 'b,c,d)\og 2 (p(a,b,c,d)) 
H(ABCD) 
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^2 -p'(a, b, c, d, r) log 2 (^p(a, b, r, d)) 

a,b,c,d.r d 

= XX _ -f'( a ' b ' c ' d ' r ) lo g2(X! p ( a ' & ' r ' rf )) 

a,b,r c.d d 

= ^2^2-p(a,b,r,d)\og 2 C^2p(a,b,r,d)) 

a,b.r d d 

= H(ABC) 



X -P'( a > b > C > d > r ) lo S2(X P ( a ' & ' C ' d ^ 

a,b,c,d,r c,d 

= XX -p'( a ,b,c,d 7 r)\og 2 (^2p(a,b,c,d)) 

a,b. c,d.r c.d 

- ^2^2-p{a,b,r,d)log 2 (^2p(a,b,c,d)) 

a, 6, rf,r c,d 

= H(AB) 

Therefore, H(ABCDR) = H(ABCD)+H(ABC)-H(AB) 
as desired. ■ 



III. The Zhang-Yeung Inequality 

Zhang and Yeung were the first to discover non-Shannon 
inequalities. Their original version of the inequality took the 
following form. 

2/(C;L») < I(A;B) + I(A;CD) + 3I(C;D\A)+I{C;D\B). 

Their proof used two copy lemmas. Here we give a proof 
that is not necessarily shorter, but it uses only one copy. We 
will show below that the Zhang-Yeung inequality is the only 
inequality that results from one instance of the copy lemma. 

We will write down the proof twice. The first proof makes 
it easier for the reader to see at a glance, how the inequality 
is achieved by applying Shannon inequalities after adding an 
auxiliary variable. However, it also takes longer to check it for 
accuracy since it requires tedious expansions of conditional 
informations into entropies. The second proof is easier to 
check for accuracy, but harder to get the general idea of the 
proof. Future proofs will be given only the first way. 

We also write the inequality in a different, but equivalent 
form. This is so it will match the pattern of the other 
inequalities that we have found. The equivalence of the two 
forms can be seen by swapping C f-> A, B <-> D and rewriting 
in terms of entropies. 

Theorem 3 (Zhang-Yeung Inequality): Let A, B, C, D be 
random variables. Then 

I(A;B) < 2I(A;B\C)+I(A;C\B)+I(B;C\A) 
+ I(A;B\D)+I(C;D) 

Proof: (A) By expanding mutual informations into en- 
tropies and canceling terms, one can verify the following 6- 



variable identity: 

I(A;B) 

+ I(C;R\A) (S) 

+ I(C;D\R) (S) 

+ I(AB;R\CD) (S) 

+ I{D-R\B) (S) 

+ I(A;B\RD) (S) 

+ I(D;R\A) (S) 

+ I(R]C\B) (S) 

+ I(A;B\CR) (S) 

+ I(C; R\ABD) (S) 
= 2I{A; B\C) + I(A; C\B) + I(B; C\A) 
+ I(A;B\D)+I(C;D) 

+ 2I(CD; R\AB) (C2) 

+ I{A;B\R) - I(A;B\C) (CI) 

+ I{A;R\B) - I(A;C\B) (CI) 

+ I(B; R\A) — I(B; C\A) (CI) 



Each of the terms marked (S) is a conditional mutual informa- 
tion and is therefore nonnegative. Thus, if the terms marked 
(S) are erased and the "=" is replaced by "<", then we obtain 
a 5-variable Shannon-type inequality. By the Copy Lemma we 
may choose R to be a D-copy of C over AB. Then, the term 
marked (C2) is zero by condition (C2), and each of the terms 
marked (CI) equals zero by condition (CI). ■ 
In order to read the second proof, it will be convenient to 
review some basic properties of entropies. Each of these can 
be verified by rewriting in terms of entropies, and applying 



Shannon's inequality (O, if necessary. 

I(A;B) = H(A)-H(A\B) (8) 

I{A;B\C) = H{A\C) - H{A\BC) (9) 

I(A;BC\D) = I{A;C\BD)+I{A;B\D) (10) 

H(A\BC) < H(A\B) < H(AC\B) (11) 

I(A;B\CD) < I(A;B\C) < I(AD;B\C) (12) 

It will also be convenient to note that property (C2) can be 
rewritten as 

H(R\AB) = H(R\ABCD). (13) 
By combining this with ( TTOb and ( fL2l it follows that 

I{A;C\B) =I(AR;C\B) (14) 

and 

I{B;R\A)=I(BD;R\A). (15) 
Proof: (B) Let R be a D-copy of C over (A, B). Then: 

(I).- 

2I(A;B\C) 

= I(A;B\C)+I{A;B\R) (CI) 
= H(A\C)-H(A\BC)+I(A;B\R) ® 
= H{A\C)-H{A\B)+I(A-C\B)+I{A;B\R) © 
= H(A\C)-H(A\B)+I(AR;C\B)+I(A;B\R) (O 

> h{a\c)-h{a\b)+i\a-c\br)+i\a-b\r) (O 
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= H(A\C)-H(A\B)+I(A;BC\R) <[[0) 

> H(A\C)-H(A\B)+I(A;C\R) O 

(II): 

I(A; C\B)+I(B; C\A)+I(A; B\D) 

= I(A-R\B)+I{B-R\A)+I{A-B\D) (CI) 

= I(A;R\B)+I(BD;R\A)+I(A;B\D) g3J 

> 7(i4;iJ|B)+J(B;ie|i4D)+J(i4;B|D) O 
= J(A;iJ|B)+J(Aii;B|D) flO) 

> I(A;R\B)+I(R;B\D) O 
= -H(R\AB)+H(R\B)+I(R;B\D) © 

> -H(R\AB)+hIr\BD)+I(R;B\D) £□} 
= -fr( J R|AB)+fl"(ii|D) © 
= -H(R\ABCD)+H(R\D) (O 

> -H(R\CD)+H(R\D) (HQ 
= I(R;C\D) © 
= H{C\D)-H(C\DR) © 

> H(C|D)-lT(C|iJ) CCD 

2IL4; C|S)+7(S; C|A) 

+J(i4;B|D)+J(C; £>) 

> 2/(A;B|C)+F(C|£))-i/(C|iJ)+/(C;D) (II) 
= 2/(A;B|C)+ff(C)-.ff(C|i2) © 

> #(A|C)-iJ(A|B)+/(A;C*|i?) 
+H(C)-H(C\R) (I) 

= fl"(A|C)-fl'(A|B)+ff(C)-ff(C|AR) © 

= -F(^| J B)+iJ( J 4C)-iJ(C|Ai?) © 

> -H(A\B)+H(AC)-H(C\A) CCD 
= © 
= © 



IV. Search Methodology 

In this section we will describe the search method used to 
generate new inequalities. 

We assume that we have a present list of known inequal- 
ities. For example, at the start, this list is just the Shannon 
inequalities. We consider the set of vectors, listed in the order 

(H(A),H(B),H(AB),H(C),H(AC),H(BC), 
H(ABC), H(D),H(AD),H(BD), H(ABD), 
H{CD), H(ACD) 1 H(BCD), H(ABCD)) (16) 

that satisfy the current list of inequalities. Since all of the 
inequalities are homogeneous, this set of vectors is closed 
under multiplication by positive constants. This set of sat- 
isfying vectors forms a region that is a polytopic cone in 
fifteen dimensions. An alternate way to view this space is to 
hold H(ABCD) at a fixed value of 1, and then consider the 
satisfying vectors (H(A), . . . , H(BCD)) which now form a 
polytope in R 14 . 

Using software, such as Fukuda's Cddlib (4), we can 
generate a list of extreme rays for the current region. Two 
rays are called equivalent if one can be obtained from the other 
under a permutation of the variables A, B, C, D. For example, 



000111111111111 is equivalent to 011001111111111 since 
the second one can be obtained from the first by swapping 
B and C. At the start of the process, when we just have 
the Shannon inequalities, there are twelve extreme rays, up 
to symmetry: 

000000000000000 
000000011111111 
000111111111111 
011111111111111 
111111111111111 
011112211222222 
112122212221222 
112122212222222 
212122212222222 
112122312232333 
112122323333333 
224233423343444 

Next, generate a list of possible one-variable copy specifi- 
cations that will create a new variable R. Again with just one 
from each symmetry family these are: 



27 


8 


7 


36 


12 


3 


39 


14 


1 


63 


4 


3 


66 


G 


1 


75 


2 


1 



Each of these codes up an instance of the Copy Lemma of 
the form "R is an A-copy of Y over Z". The first column is 
just an index and is irrelevant. The second and third columns 
code the Y and Z respectively. These variables are coded in 
binary, combining 1 = A, 2 = B, 4 = C, 8 = D. So the 
line labeled by 27 gives us that Y is D and Z is ABC, while 
the line labeled 36 gives us that Y is CD and Z is AB. The 
variable X can be deduced from the other two. For each of 
the variables A, B, C, D, if it does not appear in Z and is 
not equal to Y then it is included in X. Then the above reads 
as: 



27 


R 


is 


a 


copy of D over ABC 


36 


R 


is 


a 


CD-copy of (CD) over AB 


39 


R 


is 


a 


BCD-copy of (BCD) over A 


63 


R 


is 


a 


D-copy of C over AB 


66 


R 


is 


a 


BCD-copy of (BC) over A 


75 


R 


is 


a 


CD-copy of B over A 



(Some possibilities were elimintated in advance: a last copy 
over no variables can never produce any new inequalities, and 
we may assume that no variable appears in both Y and Z.) 

Now go through a full search, testing each of the copy 
specifications against all of the extreme rays (including all 
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symmetric forms) to see whether any contradictions are ob- 
tained. In the current example, the results of the run are: 



27 

36 
39 



8 

12 
14 



7 
3 
1 



-PASSED- 
-PASSED- 
-PASSED- 



— 223233424343444 
Inequality: -1 -2 3 -2 3 3 -4 -1 1 1 -1 

63 4 3 

66 6 1 -PASSED- 
75 2 1 -PASSED- 

The 'PASSED' entries are those copy specifications which 
did not contradict any of the extreme rays. The remaining copy 
specification, "R is a D-copy of C over AB", was incompatible 
with the extreme ray 223233424343444 (and possibly 
with others; the search stops for a given copy specification 
when one extreme ray fails). In other words, the equations 
forcing an entropy vector to lie on this extreme ray, namely 



H{A) 


= 2H{ABCD)/A 


(17) 


H(B) 


= 2H(ABCD)/A 


(18) 


H(AB) 


= 3H(ABCD)/A 


(19) 


H(C) 


= 2H(ABCD)/A 


(20) 


H{AC) 


= 3H(ABCD)/A 


(21) 


H{BC) 


= 3H(ABCD)/A 


(22) 


H(ABC) 


= AH{ABCD)/A 


(23) 


H(D) 


= 2H(ABCD)/A 


(24) 


H(AD) 


= AH(ABCD)/A 


(25) 


H(BD) 


= 3H(ABCD)/A 


(26) 


H(ABD) 


= AH(ABCD)/A 


(27) 


H(CD) 


= 3H(ABCD)/A 


(28) 


H(ACD) 


= AH(ABCD)/A 


(29) 


H(BCD) 


= AH(ABCD)/A 


(30) 



together with the copy specification requirements coming from 
the Copy Lemma, when R is a D-copy of C over AB, 



H(R) 
H(AR) 
H(BR) 
H(ABR) 
I(CD;R\AB) 



H{C) 

H(AC) 

H(BC) 

H(ABC) 





(31) 
(32) 
(33) 
(34) 
(35) 



and the Shannon inequalities, force all of the entropies to be 
0. 

^From this one can generate a new inequality. The software 
has already gone through this process to produce the inequality 

-H(A) - 2H(B) + 3H(AB) - 2H(C) 
+3H(AC) + 'SH(BC) - AH(ABC) - H(AD) 

+H(BD) +H(CD) - H(BCD) >= 0, 

To explain this process in more detail, here is how one could 
arrive at the same inequality using a program called ITIP 



£[2). ITIP is a MATLAB |7j] program for testing Shannon 
inequalitites. It was written in made available by Raymond 
Yeung and Ying-On Yan. One feeds ITIP an entropy inequality 
followed by a list of linear constraints. ITIP returns "True" if 
the inequality follows from the constraints and the Shannon 
inequalities. Otherwise it returns "Not provable by ITIP". 
Start with: 

ITH>('H(ABCD) <= 0',... 

'AH(A) = 2H(ABCD)\ i AH(B) = 2H(ABCD)\... 
l AH(AB) = 3H(ABCDy;AH(C) = 2H(ABCD)\... 
l AH(AC) = 3H{ABCDy:AH(BC) = 3H(ABCD)\... 
'AH(ABC) = AH{ABCDy;AH{D) = 2H{ABCD)\... 
'AH(AD) = AH(ABCDy:AH(BD) = 3H(ABCDY,-~ 
'AH(ABD) = AH{ABCD)\... 
'AH(CD) = 2,H{ABCD)\... 
'AH(ACD) = AH{ABCD)\... 
'AH(BCD) = AH(ABCDY,'I(R;CD\AB) = 0',... 

'H{R) = h(cy:h(ar) = H(ACY,... 

'H(BR) = H(BCY:H(ABR) = H(ABCY) 
True 

Note that each inequality and each constraint is contained in 
single quotes, and they are separated by commas. The ellipses 
at the end of each line is part of the MATLAB syntax and 
merely denotes that the command is continued on the next 
line. 

This first run of ITIP merely justifies what has already been 
claimed, namely that ITIP can prove H(ABCD) <= from 
equations ( TPTI i to (135V 

Next, we test (by trial and error) whether some of these 
constraint equations can be eliminated. We find that, in this 
case, equations dTD i, d24l) . d27l i. < f29b are not necessary. The 
inquiry to ITIP looks like: 

mP('H(ABCD) <= 0',... 

'AH(B) = 2H{ABCDY;AH{AB) = ZH{ABCD)\... 
'AHiC) = 2H{ABCDY:AH(AC) = 3H(ABCD)\... 
'AH(BC) = 3H(ABCDY,... 
'AH(ABC) = AH{ABCD)\... 

l AH(AD) = AH{ABCDY:AH{BD) = 3H(ABCDY 
'AH(CD) = 'iH(ABCDY 

'AH(BCD) = AH(ABCDY,'I{R;CD\AB) = 0',... 
'H(R) = H{C)';H{AR) = H(AC)\... 
'H(BR) = H(BC)\ l H(ABR) = H(ABCY) 
True 

Next, one can eliminate each equation constraint by adding 
an appropriate multiple of it (determined perhaps by trial and 
error; we modified ITIP to output some additional information 
that would be helpful for this) to the right-hand side of the 
inequality. For example, for the first remaining constraint < fT~8l ). 
we first try using a multiple of zero: 

1TU>CH(ABCD) <= -0H(B) + m{ABCD)' 
'AH(AB) = 3H(ABCDY,... 

'AH(C) = 2H{ABCDY:AH{AC) = 3H{ABCD)\... 
'AH(BC) = 3H(ABCDY 
l AH(ABC) = AH{ABCD)\... 

'AH(AD) = AH(ABCDY:AH(BD) = 3H(ABCDY 
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'AH {CD) = 3H{ABCD)\... 
1 AH {BCD) = AH {ABC D)',' I {R; CD\AB) = 0',... 
H{R) = H{C)','H{AR) = H{AC)\... 
'H{BR) = H{BCy;H{ABR) = H{ABCy) 
Not provable by ITIP 

This of course fails since it was already determined that this 
equation was necessary. Then we gradually raise the multiple 
until we get a success. Thus, 

YYIP('H{ABCD) <= —8H{B) + AH{ABCD)' 
'AH(AB) = 3H(ABCD)' 

'AH{C) = 2H{ABCD)\'AH{AC) = 3H{ABCD)\... 
'AH{BC) = 3H{ABCD)\... 
'AH {ABC) = AH{ABCD)\... 

'AH {AD) = AH{ABCD)','AH{BD) = 3H{ABCD)\... 
'AH {CD) = 3H{ABCD)\... 
'AH {BCD) = AH {ABC D)',' I {R; CD\AB) = 0\... 
H{R) = H{C)','H{AR) = H{AC)',... 
'H(BR) = H{BC)\'H{ABR) = H{ABC)') 
True 

We repeat this for the next equation, finding 

ITIP('H{ABCD) <= —8H{B) + AH {ABC D)',... 
'+12H{AB) - 9H{ABCD)',... 
'AH{C) = 2H{ABCD)\'AH{AC) = 3H{ABCD)',... 
'AH{BC) = 3H{ABCD)' 
'AH {ABC) = AH {ABC D)',... 

'AH {AD) = AH{ABCD)','AH{BD) = 3H{ABCD)',... 
'AH {CD) = 3H{ABCD)\... 
'AH {BCD) = AH{ABCD)\'I{R; CD\AB) = 0',... 
H{R) = H{C)\'H{AR) = H{AC)\... 
'H(BR) = H{BCy,'H{ABR) = H{ABC)') 
True 

After passing through all of the equations not involving R, 
we get: 

ITIP(['H{ABCD) <= —8H{B) + AH{ABCD)' 
'+12H{AB) - 9H{ABCD)\... 

'-8H{C)+AH{ABCD) + 12H{AC) - 9H{ABCD)',... 
'+12H{BC) - 9H{ABCD)\... 
'-16H{ABC) + 16H{ABCD)',... 

'-8H{AD) + 8H{ABCD) + 8H(BD) - 6H(ABCD)\... 
'+8H{CD) - 6H{ABCD)',... 

'-8H{BCD) + 8H{ABCD)'],'I{R; CD\AB) = 0',... 
H{R) = H{C)\'H{AR) = H{ACy 
'H(BR) = H{BCy,'H{ABR) = H{ABC)') 
True 

Note that brackets are used to enclose the first inequality 
when it does not fit on a single line. Thus ITIP gives a proof 
of a new inequality under the copy conditions. We collect like 
terms and divide through by the common divisor, 4, to get: 

ITIP(['-2F(B) + 3H(AB) - 2H{C) + 3H{AC)\... 
'+3H{BC) - AH {ABC) - 2H{AD) + 2H{BD)\... 
'+2H{CD) - 2H{BCD) >= 0'],... 
'I{R; CD\AB) = 0','H{R) = H{C)',... 
H{AR) = H{AC)\'H{BR) = H{BC)\... 
l H{ABR) = H(ABCy) 



True 

For the last step, we see if this inequality can be strength- 
ened by adding one or more Shannon terms to the right-hand 
side. This final inequality simplifies to the form 

-H{A) - 2H{B) + 3H(AB) - 2H{C) 
+3H(AC) + 3H{BC) - AH(ABC) - H{AD) 

+H{BD) + H{CD)-H{BCD) >= 0, 

mentioned earlier. 

One can express this inequality (which is in fact the Zhang- 
Yeung inequality) in the form 

I{B-C) <= 2I{B;C\A)+I{B;A\C)+I{C;A\B) 
+I{B;C\D)+I{A; D). 

Also, one can do further ITIP work to extract an intelligible 
proof of this inequality from the Copy Lemma. 

Once one has this inequality, one can produce a new list of 
extreme rays up to symmetry: 

000000000000000 (36) 

000000011111111 

000111111111111 

011111111111111 

111111111111111 

011112211222222 

112122212221222 

112122212222222 

212122212222222 

112122312232333 

112122323333333 

324456645565666 

336244645565666 

336345635465666 

336455645565666 

A test of this list against the remaining one-variable copy 
specifications shows that there are no further contradictions 
- the Zhang-Yeung inequality is all that one can get using one 
copy variable. 

To get more inequalities, we try using two copy variables 
but just one copy step. Then RS is a W^-copy of {X){Y) over 
Z. The possible copy specifications are: 



18630 


4 


8 


3 


18666 


6 


10 


1 


18675 


6 


8 


1 


18837 


2 


12 


1 


19566 


2 


4 


1 



As before, the first column is an index and is irrelevant. The 
variables X, Y, and Z are coded by the second, third, and 
fourth columns respectively. The variable W is deduced from 
the others by including any of A, B, C, D which do not appear 
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in Z and are not equal to X or Y. Thus, the above translates 
as: 

RS is a copy of CD over AB 

RS is a BCD-copy of (BC)(BD) over A 

i?5 is a BC-copy of (BC)£) over 4 

RS is a CD-copy of B(CD) over A [redundant] 

i?5 is a D-copy of i?C over A 

It turns out that these also do not give any more than the 
Zhang-Yeung inequality. 

For two copy variables in two steps, there is a much longer 
list of possibilities (753 of them). A first pass against the 
Shannon extreme rays reduces this list to 198, and then a run 
against the Zhang-Yeung extreme rays shows that only 15 of 
the copy specifications give contradictions (new inequalities): 

— 445355625664666 

Inequality: -7 -2 8 -4 9 5 -9 3 -2 3 -4 
4636 4 3 1 28 
R is a D-copy of C over AB 
S is a £?-copy of A over CDR 

— 425354645665666 

Inequality: -7 -1 7 -4 9 5 -9 -1 4 -2 3 -4 
4716 4 3 4 25 
R is a D-copy of C over AB 
S is a £?-copy of C over ADR 

— 425354635466666 

Inequality: -6 -3 7 -2 7 5 -8 -1 4 3 4 -2 
4788 4 3 2 25 

— 245345634566666 

Inequality: -3 -6 7 -3 6 8 -9 -1 3 4 -4 -2 
4798 4 3 1 24 



Inequality: -3 -4 7 -6 7 10 -1 1 2 -2 2 -2 
6094 4 3 1 20 

— 345245645566666 

Inequality: -5 -5 8 -1 6 6 -9 2 2 -2 -2 
6156 4 3 4 19 

— 345455624665666 

Inequality: -3 -4 7 -6 7 10 -1 1 2 -2 2 -2 
6174 4 3 4 17 

— 245345634566666 

Inequality: -3 -5 6 -2 5 6 -7 -1 3 4 -4 -2 
11358 4 3 16 9 

— 445355635566666 

Inequality:-8 -9 14 -1 7 8 -1 1 -1 6 6 -8 -3 
12069 4 3 24 3 

— 345455624665666 

Inequality: -3 -3 6 -5 6 8 -9 2 -2 2 -2 
12654 4 3 16 5 

— 345245645566666 

Inequality: -5 -5 8 -1 6 6 -9 2 2 -2 -2 
12717 4 3 20 3 

One can now continue the iteration, adding these new 
inequalities to the known list, producing a new set of extreme 
rays, re-running them against the 15 remaining copy specifica- 
tions, and repeating until no contradictions remain. The result 
(after weeding out redundant inequalities that can be deduced 
from the others) is the list of six two-copy-variable inequalities 
given in our previous paper. 



— 445355625664666 

Inequality: -17 -2 16 -8 21 7 -17 -2 9 -4 9 -12 
5364 4 3 8 21 

— 445355635566666 

Inequality: -7 -7 11 -1 6 6 -8 -1 6 6 -8 -3 
5508 4 3 8 19 

— 245345634566666 

Inequality: -3 -6 7 -3 6 8 -9 -1 3 4 -4 -2 
5526 4 3 8 17 

— 324455645665666 

Inequality: -6 -1 7 -5 9 6 -10 2 -2 2 -2 
6084 4 3 2 21 

— 3345455624665666 



V. Inequalities Using Two Copies 

In our conference paper, we gave six additional inequalities 
that used two copy lemmas to prove. For this article, we will 
rewrite these inequalities in a different form that matches our 
other inequalities. We summarize as follows: 



2I(A;B) < aI(A;B\C)+bI(A;C\B)+d(B;C\A) 
+ dI(A;B\D) + eI(A;D\B) + fI{B;D\A) 
+ gI(C;D) 



for each of the follwing values of (a, b, c, d, e, /, g). 




(a,b, c,d,e,f,g) 


= (5,3,1,2,0,0,2) 


(37) 


(a,b, c,d,e,f,g) 


= (4,2,1,3,1,0,2) 


(38) 


(a,b, c,d,e,f,g) 


= (4,4,1,2,1,1,2) 


(39) 


(a,b, c,d,e,f,g) 


= (3,3,3,2,0,0,2) 


(40) 


(a,b, c,d,e,f,g) 


= (3,4,2,3,1,0,2) 


(41) 


(a,b, c,d,e,f,g) 


= (3,2,2,2,1,1,2) 


(42) 
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These are the only additional inequalities that are obtained 
from this technique using two copy lemmas. As it turns 
out, all of these inequalities except for (138) have since been 
superseded by other inequalities. We give here a proof only 
for (08). 

Lemma 4: The following is a 5-variable information in- 
equality: 

I (A; B) < I(A;B\C)+I(A;B\D)+I(C;D) 

+ I(A;D\R)+I(A;R\D)+I(D;R\A). 

Proof: By expanding mutual informations into entropies 
and cancelling terms, one can verify the following 6-variable 
identity: 



I(A;B) 

+ I(A;B\CS) (S) 

+ I(B;S\A) (S) 

+ I(B;S\D) (S) 

+ I(A;D\BS) (S) 

+ I(C;D\S) (S) 

+ I(C;S\A) (S) 

+ I(C;S\B) (S) 

+ I{ABR-S\CD) (S) 

+ I(CR;S\ABD) (S) 

+ I(DR;S\ABC) (S) 
= I(A; B\C) + I{A- B\D) + I(C: D) 
+ I(A; D\R) + I(A; R\D) + I(D; R\A) 

+ 3I{S;BCR\AD) (C2) 

-(H(S)-H(R)) (CI) 

+ 2(H(SA) - H(RA)) (CI) 

+ 2(H(SD) — H(RD)) (CI) 

- 3(H(SAD) - H(RAD)) (CI) 



Each of the terms in the lines marked (S) is a conditional 
mutual information and is therefore nonnegative by Shannon's 
inequality (f2). Thus, if the terms marked (S) are erased and the 
"=" is replaced by "<", then we obtain a 6-variable Shannon- 
type inequality. By the Copy Lemma, we may choose the 
random variable S such that S is a BC-copy of R over AD. 
Then the term marked (C2) is zero by condition (C2), and 
each of the terms marked (CI) equals zero by condition (CI). 

■ 

Theorem 5: The following is a 4-variable information in- 
equality: 

2I(A;B) < 4I(A;B\C)+2I(A;C\B)+I(B;C\A) 
+ 3I{A;B\D)+I{A;D\B)+2I(C;D). 

Proof: By expanding mutual informations into entropies 
and cancelling terms, one can verify the following 5-variable 



identity: 

2I(A;B) 

+ I{A-B\CR) (S) 

+ 3I{A;B\DR) (S) 

+ I(C;D\R) (S) 

+ I(D;R\ABC) (S) 

+ 3I{D;R\B) (S) 

+ I(AB; R\CD) (S) 

+ 3I{C;R\ABD) (S) 

+ I(C;R\A) (S) 

+ I(C;R\B) (S) 



+ [-I(A; B) + I(A; B\C) + I(A; B\D) 
+ I(C; D) + I(A; D\R) + I(A; R\D) + I(D; R\A)} (L) 
= AI{A; B\C) + 2I(A; C\B) + I(B; C\A) 
+ 3I{A; B\D) + I(A: D\B) + 2I(C; D) 



+ 5I(R;CD\AB) (C2) 

-2(H(R)-H(C)) (CI) 

+ 3(H (RA) - H (CA)) (CI) 

+ A(H(RB) -H{CB)) (CI) 

- 5(H(RAB) - H{CAB)) (CI) 



The expression enclosed in brackets "[]" immediately before 
the "=" sign and marked with (L) is nonnegative by Lemma 
H] Each of the terms in the lines marked (S) is a conditional 
mutual information and is therefore nonnegative. Thus, if the 
terms marked (S) are erased and the "=" is replaced by "<", 
then we obtain a 5-variable Shannon-type inequality. By the 
Copy Lemma, we may choose R such that R is a D-copy of C 
over AB. Then the term marked (C2) is zero by condition (C2), 
and each of the terms marked (CI) equals zero by condition 
(CI). ■ 

VI. Inequalities Using Three Copy Variables with 
at most Two Copy Steps 

In this section and the next we will present all of the 
information inequalities that can be proved from this technique 
using three copies. Some of these inequalities have been 
superseded by other inequalities, and won't be listed here. 
In this section we will concentrate on those that involve only 
two instances of the Copy Lemma. We will start out by giving 
the proof of the first inequality in full detail. The proofs are 
all very similar. Therefore, in order to save space, as we 
go further down the list we will gradually take shortcuts, 
eventually giving only abbreviated proofs. However, enough 
detail will given in these outlines so that the full proofs can 
be reconstructed. 

Theorem 6: The following is an information inequality 

aI(A;B) < bI(A; B\C) + d(A;C\B) + dI(B;C\A) 
+ eI(A; B\D) + fI(A; D\B) + gI(B; D\A) 
+ hI(C;D)+U(C;D\A) 
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for each of the following values of (a, b, c, d, e, f, g, h, i). To see this, verify the following 7-variable identity by expand- 
ing into entropies and canceling: 

2I(A;B) 



(3, 


4, 


4- 


4, 


3, 


1. 


1, 


3. 


0) 


(43) 


(3. 


9, 


6, 


1. 


3, 


0. 


0, 


3. 


0) 


(44) 


(3. 


4, 


6, 


G, 


3, 


0. 


0, 


3. 


0) 


(45) 


(2. 


3, 


3. 


1, 


5, 


2. 


0, 


2. 


0) 


(46) 


(3. 


4, 


3. 


3, 


3, 


3- 


3, 


3. 


0) 


(47) 


(3. 


6, 


3. 


1. 


G, 


3. 


0, 


3. 


0) 


(48) 


(4. 


5, 


8, 


8. 


4. 


1. 


1- 


4. 


0) 


(49) 


(2. 


4, 


2- 


1. 


2. 


0. 


0, 


2- 


3) 


(50) 


(4. 


5, 


5. 


5. 


4. 


4. 


4. 


4. 


0) 


(51) 


(2. 


3, 


3. 


2, 


2, 


0. 


0, 


2. 


0) 


(52) 


(3. 


7, 


5, 


1. 


3, 


1- 


1, 


3. 


0) 


(53) 


(4. 


6, 


4. 


3, 


4, 


2- 


1, 


4. 


0) 


(54) 


(2. 


5, 


2- 


1. 


2. 


0. 


0, 


2- 


0) 


(55) 


(2. 


4, 


3. 


1. 


2. 


0. 


0, 


2- 


0) 


(56) 


(2. 


4, 


1. 


2, 


2. 


3. 


0, 


2- 


0) 


(57) 


(2. 


4, 


2- 


1. 


2. 


4. 


1, 


2- 


0) 


(58) 



+ I(CR; S\A) 


(S) 


+ I{CR;S\B) 


(S) 


+ I{DR;S\A) 


(S) 


+ I(DR;S\B) 


(S) 


+ I(CD\ST\ABR) 


(S) 


+ I(A;B\CRS) 


(S) 


+ I(A]B\DRS) 


(S) 


+ I{AB;S\CDR) 


(S) 


+ I(C;D\RS) 


(S) 


+ I{D;S\ABCR) 


(S) 


+ I{C;S\ABDR) 


(S) 


+ I(CR;T\A) 


(S) 


+ I(CR;T\B) 


(S) 


+ I(DRT\A) 


(S) 


+ I{DR;T\B) 


(S) 


+ I{A;B\CRT) 


(S) 


+ I(A; B\DRT) 


(S) 


+ I{C;D\RT) 


(S) 


+ I(AB;T\CDR) 


(S) 


+ +3I(CDR;T\ABS) 


(S) 


+ I(CDR;S\ABT) 


(S) 


+ I(S;T\R) 


(S) 


+ I{AB;R\ST) 


(S) 


+ I(CR; S\ABT) 


(S) 


+ I(DR\ S\ABT) 


(S) 


+ I{C; ST\ABDR) 


(S) 


+ I(D;ST\ABCR) 


(S) 


= I(AB; R) + I(A; B\C) + I(A; B\D) 





+ 21 (A; B\CR) + 2I{A; B\DR) + I(A; C\B) 
+ I(A; D\B) + I(B- C\A) + I(B; D\A) 

Proof: Proof of @3): We first show the following 5- + I{C; D) + 2I(C; D\R) 

variable information inequality: + 7 1 (ST; CDR\AB) (C2) 

- (H(ST) - H(CD)) (CI) 
+ 2(H(SA) - H (CA)) (CI) 
+ 2(H(TA) — H(DA)) (CI) 
+ 2(H(SB) - H(CB)) (CI) 
+ 2(H(TB) — H(DB)) (CI) 

- 3(H(SAB) - H(CAB)) (CI) 

- 3(H(TAB) - H(DAB)) (CI) 



21 (A; B) < I(AB;R)+I(A;B\C)+I(A;B\D) 

+21 (A; B\CR) + 21 (A; B\DR) + I (A; C\B 



Each of the terms in the lines marked (S) is a conditional 
mutual information and is therefore nonnegative. Thus, if the 
+I(A; D\B) + I(B; C\A) + I(B; D\A) terms marked (*S) are erased and the "=" is replaced by 

+J(C; D) + 21 (C; D\R). (59) "<", then we obtain a 7-variable Shannon-type inequality. By 
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the Copy Lemma, we may choose ST such that ST is a R- 
copy of CD over AB. Then the term marked (C2) is zero by 
condition (C2), and each of the terms marked (CI) equals zero 
by condition (CI). 

With (|59b verified, we now continue with the proof of 
( |43] >. By expanding mutual informations into entropies and 
cancelling terms, one can verify the following 5-variable 
identity: 



3I(A;B) 

+ 2I(C;R\A) (S) 
+ 2I(C;R\B) (S) 
+ 2I(D;R\A) (S) 
+ 2I(D;R\B) (S) 
+ 2I(AB;R\CD) (S) 
+ 2I(D;R\ABC) (S) 
+ 2I{C;R\ABD) (S) 
+ [-2I(A; B) + I(AB; R) + I(A; B\C) 
+ I(A; B\D) + 2I{A; B\CR) + 2I(A; B\DR) 
+ I{A; C\B) + I(A; D\B) + I(B; C\A) 
+ I(B;D\A)+I(C;D)+2I{C;D\R)] (L) 
= AI(A; B\C) + 4I(A; C\B) + 4I(B; C\A) 
+ 3I(A; B\D) + I(A; D\B) + I(B; D\A) 
+ 3I(C;D) 

+ 6I(R;CD\AB) (C2) 

- (H(R) - H(C)) (CI) 
+ A(H(RA) - H(CA)) (CI) 
+ 4(H(RB) - H(CB)) (CI) 

- 7(H(RAB) - H(CAB)) (CI) 



The expression in brackets "[]" marked (L) is nonnegative by 
Lemma 11.1. Each of the terms in the lines marked (S) is a 
conditional mutual information and is therefore nonnegative. 
Thus, if the terms marked (S) are erased and the "=" is 
replaced by "<", then we obtain a 5-variable Shannon-type 
inequality. By the Copy Lemma, we may choose R such that 
R is a D-copy of C over AB. Then the term marked (C2) is 
zero by condition (C2), and each of the terms marked (CI) 
equals zero by condition (CI). This finishes the proof of (1431 . 

Proof of (l44l : The proof of each of the remaining inequali- 
ties has the same form as the proof of (143V Therefore, we will 
gradually take more and more shortcuts. For each inequality 
there are two equations. The first equation proves a lemma, 
and the second proves the inequality. Each line marked with 
(S), (L), (CI), or (C2) can be ignored because it is a Shannon 
inequality, an instance of the lemma, or it follows from (CI) or 
(C2) resp. The inequalities are then formed by ignoring these 
lines and turning the "=" sign into a "<" sign. 

We now give an outline of the proof of (1441) : 



2I{A-B) 

+ I(A; B\CT) + I(A; B\DT) + I(A; B\RT) (S) 

+ I(A; R\ST) + I (A; T\CD) + I(ACDR; T\BS) (S) 

+ I{C] D\T) + I(C; T\B) + I(CRS; T\ABD) (S) 

+ I(CDS; T\ABR) + I(C; T\ARS) + I(B; S\T) (S) 

+ I{B; T\ACDRS) + I(D; RS\ACT) + I(D; T\A) (S) 

+ I(D; T\B) + I{DRS; T\ABC) + I(R; T\A) (S) 

+ I{R;T\B) + I(S;T\A) + I(S;T\R) (S) 
= I(A; B\C) + I(A; B\D) + 3I(A; B\R) 
+ 2I(A; R\B) + I(A; R\S) + I(C; D) 
+ I(B; R\A) + I(B; S) + I(D; RS\AC) 

+ 5I(T;CDRS\AB) (C2) 

- 2{H{T) - H(R)) + 3(H(TA) - H(RA)) (CI) 
+ A(H(TB) - H(RB)) - 5(H(TAB) - H(RAB)) (CI) 

where T is a CDS-copy of R over AB. Then, 
3/L4; B) 

+ 6I(A; C\BR) + I(A; B\DR) + 6I(B; R\C) (S) 

+ I(D; R\A) + I(C; D\R) + /(AB; R\CD) (S) 

+ I(D; R\B) + J(C; R\ABD) + I(B; RS\ACD) (S) 

+ 6I(D: R\ABC) + 7(C; S\B) + I(AR; B\CS) (S) 

+ 8I(BD;S\ACR)+I(D;RS\ABC) (S) 

+ [-2I{A;B)+I(A;B\C)+I{A:B\D) (L) 

+ 3I(A; B\R) + 2I(A; R\B) + I(A; R\S) (L) 

+ I(C; D) + I(B; R\A) + I(B; S) (L) 

+ I(D;RS\AC)} (L) 
= 9I(A; B\C) + 6I(A; C\B) + I(B: C\A) 
+ 3I(A;B\D) + 3I(C;D) 

+ 10I(RS;BD\AC) (C2) 

-A{H{R)-H{B)) + {H{RS)-H{BD)) (CI) 

+ 5(H(RA) - H(BA)) + {H{SA) - H{DA)) (CI) 

- (H(RSA) - H(BDA)) + 7(H(RC) - H{BC)) (CI) 

- (H(SC) - H{DC)) - 8(H(RAC) - H(BAC)) (CI) 



where RS is a copy of BD over AC. This finishes the proof 
of <|44). 

Shortened Proof of 04): In the proof of the above lemma 
there are some lines marked (S). These lines make up a 
Shannon inequality. On the other side of the equation, there 
is a line marked (C2). Except for the coefficient, this comes 
directly from the Copy Lemma (Lemma |2). Finally, there are 
some lines marked (CI). These lines are zero by the Copy 
Lemma, and are easily deduced from the rest of the equation. 
We will now rewrite the proof in a more abbreviated form: 



12 



Let T be a CD S-copy of R over AB. Let (L) be non- 
negative expression 

- 2I(A; B) + I(A; B\C) + I(A; B\D) + 3I(A; B\R) 
+ 3I(A; B\R) + 2I(A; R\B) + I(A; R\S) + 7(C; D) 
+ I(B; R\A) + I(B; S) + I(D; RS\AC) , 

which is proved using the Shannon inequality < 

+ I(A; B\CT) + I(A; B\DT) + I{A] B\RT) 
+ I(A; R\ST) + I{A- T\CD) + I{ACDR; T\BS) 
+ 7(C; D\T) + 7(C; T\B) + I{CRS; T\ABD) 
+ I(CDS; T\ABR) + 7(C; T\ARS) + I(B; S\T) 
+ I(B; T\ACDRS) + 7(7); RS\ACT) + 7(D; T\A) 
+ I(D; T\B) + I{DRS; T\ABC) + I{R; T\A) 
+ I(R; T\B) + I(S; T\A) + I(S; T\R) , 

and a (C2) term 5I(T;CDRS\AB). 

Similarly, the main proof of inequality (l44t will be abbrevi- 
ated as follows. Let RS be a copy of BD over AC. The proof 
uses the Shannon inequality < 

3I(A; B) + 6I(A; C\BR) + I(A; B\DR) + 6I{B; R\C) 
+ I{D- R\A) + I{C; D\R) + I(AB; R\CD) 
+ I(D; R\B) + I(C: R\ABD) + I(B; RS\ACD) 
+ 6I(D; R\ABC) + I{C; S\B) + I{AR; B\CS) 
+ 8I(BD; S\ACR) + I(D; RS\ABC) 

and a (C2) term 10I(RS; BD\AC). 

Proof of <BU: Let T be a BDS-copy of R over AC. Let (L) 
be the non-negative expression 

- 2I(A; C) + 3I(A; C\R) + I(C; D) + 2I(C; R\A) 
+ I(C; S) + I(A; B\D) + I(A; R\C) + I(A; R\S) 
+ I(A;CS\B) , 

which is proved using the Shannon inequality < 

I(B: T\A) + I(CS: A\BT) + I{C\ D\T) + I(D; T\B) 
+ I(ABRS; T\CD) + I(D; T\A) + I(A; B\DT) 
+ I(CRS; T\ABD) + I{R; T\A) + I{R; T\C) 
+ I(A; C\RT) + I(BDS; T\ACR) + I(C; S\T) 
+ I(B; T\CS) + I(S; T\A) + I(DR; T\ABCS) 
+ I(S; T\R) + I(BCD; T\ARS) , 

and a (C2) term 5J(T; BDRS\AC) . Then let RS be a copy 
of AD over BC. The proof of (l45t uses the Shannon inequality 

< 

6I(A; R\B) + 6I(B; C\AR) + I(C; D\R) + I(D; R\A) 
+ I(D; R\B) + I{AB; R\CD) + I(A; B\DR) 
+ SI{D; R\ABC) + I(C; R\ABD) + I(A; S\B) 
+ I(B; R\AS) + 8I(A; S\BCR) + I(A; R\BCS) 
+ I(A; C\BRS) + 10/p; S\ABCR) 

and a (C2) term 10I(RS; AD\BC). 



Proof of (g6): Let T be a BCS-copy of R over AD. Let (L) 
be the non-negative expression 

- I(A\ D) + I(A; B\CR) + I(A; D\S) + I(A; D\R) 
+ I(A; D\BR) + 2I(A; R\D) + I(C; D\R) + I(D; R\A) 
+ I(C;S\ABR)+I(R;S) , 

which is proved using the Shannon inequality < 

I(BR; T\D) + I(CR; T\A) + I{S\ T\A) + I(S; T\D) 
+ I(A; D\ST) + I(BCR; T\ADS) + I(R; T\D) 
+ I(A; D\BRT) + I(C; T\BR) + I(A; B\CRT) 
+ 7(C; D\RT) + I(ABS: T\CDR) + I{CS; T\ABDR) 
+ I(R; S\T) + I(AB; T\RS) + I{C; S\ABRT) 
+ I(D;T\ABCRS) , 

and a (C2) term 4J(T; BCRS\AD). Then let RS be a copy of 
CD over AB. The proof of (|46| | uses the Shannon inequality 

< 

I{D; S\B) + I(A; B\DS) + 6I(C; R\ABD) 

+ I(D; RS\ABC) + 5I(D; R\B) + AI{A; B\DR) 

+ I(C; R\A) + I(C; R\B) + I{AB; R\CD) 

+ I(D; R\ABS) + 6I(D; S\ABR) + 77(C; S\ABDR) 

and a (C2) term 8I(RS;CD\AB). 

Proof of @7): Let T be a CDRS-copy of (DR) over AB. 
Let (L) be the non-negative expression 

- I(A; B) + 2I(B; DR\A) + 21 (A; DR\B) 
+ 2I(A; B\DR) + I(C; D) + I(A; B\C) 
+ I(A;B\DS)+I(R;S\D) , 

which is proved using the Shannon inequality < 

7(C; T\A) + I(C; T\B) + I(A; B\CT) + 7(C; D\T) 
+ I(ABRS; T\CD) + I(DRS; T\ABC) + I(DS; T\A) 
+ 7(7)5; T\B) + I(A; B\DST) + I(CR; T\ABDS) 
+ I(DR; T\A) + I(DR; T\B) + I(A; B\DRT) 
+ I{CS\ T\ABDR) + I{R; S\DT) + I(ABC; T\DRS) , 

and a (C2) term 57(T; CDRS\AB). Then let RS be a copy of 
CD over AB. The proof of (|47| | uses the Shannon inequality 

< 

7(7»; S\A) + I(D; S\B) + 8I(C; R\ABD) 
+ I{D- RS\ABC) + 47(7>; R\A) + 4I(D; R\B) 
+ I(C; R\A) + I(C; R\B) + I(A; B\CR) + I(C; D\R) 
+ I(AB; R\CD) + I{AB; D\RS) + 7(7); R\ABS) 
+ 7I(D; S\ABR) + I(C; S\ABR) + 9I(C; S\ABDR) 

and a (C2) term 10I(RS; CD\AB). 

Abbreviated Proof of (l47l : ^From here on the proofs will 
be further abbreviated. We begin with an abbreviation of the 
above proof. 

T: CDRS-copy of (DR) over AB. 

L: -a.b. +2b.dr.a +2a.dr.b +2a.b.dr +c.d. +a.b.c +a.b.ds +r.s.d 
SL: c.t.a +c.t.b +a.b.ct +c.d.t +abrs.t.cd +drs.t.abc +ds.t.a 
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+ds.t.b +a.b.dst +cr.t.abds +dr.t.a +dr.t.b +a.b.drt +cs.t.abdr 
+r.s.dt +abc.t.drs C2L: 5t.cdrs.ab 
RS: copy of CD over AB. 

S: d.s.a +d.s.b +8c.r.abd +d.rs.abc +4d.r.a +4d.r.b +c.r.a +c.r.b 
+a.b.cr +c.d.r +ab.r.cd +ab.d.rs +d.r.abs +7d.s.abr +c.s.abr 
+9c.s.abdr C2: 10rs.cd.ab 

Abbreviated Proof of <@8}: T: BCS-copy of R over AD. 
L: -2a.d. +a.b.c +ar.d.b +a.d.s +3a.d.r +2a.r.d +c.d. +d.r.a 
+c.s.abr +r.s. 

SL: b.t.d +ar.d.bt +c.t.a +c.t.b +a.b.ct +c.d.t +abrs.t.cd +s.t.a 
+s.t.d +a.d.st +bcr.t.ads +r.t.a +r.t.d +bcs.t.adr +a.d.rt +r.t.abc 
+cs.t.abdr +r.s.t +ab.t.rs +d.t.abcrs +c.s.abrt C2L: 5t.bcrs.ad 
RS: copy of CD over AB. 

S: d.s.b +a.b.ds +8c.r.abd +d.rs.abc +6d.r.b +6a.b.dr +c.r.a 
+c.r.b +a.b.cr +c.d.r +ab.r.cd +d.r.abs +8d.s.abr +9c.s.abdr 
C2: 10rs.cd.ab 

Abbreviated Proof of <|49): T: CDRS-copy of (CR) over 
AB. 

L: -a.b. +3a.cr.b +3b.cr.a +2a.b.cr +a.b.dr +a.b.cs +a.b.ds 
+c.d.r +c.d.s +r.s. 

SL: cr.t.a +cr.t.b +a.b.crt +dr.t.a +dr.t.b +a.b.drt +c.d.rt 
+ds.t.abcr +cs.t.abdr +abs.t.cdr +cs.t.a +cs.t.b +a.b.cst +ds.t.a 
+ds.t.b +a.b.dst +c.d.st +cr.t.abds +dr.t.abcs +abr.t.cds +r.s.t 
+abcd.t.rs C2L: 7t.cdrs.ab 
RS: copy of CD over AB. 

S: 5c.r.a +5c.r.b +d.r.a +d.r.b +ab.r.cd +lld.r.abc +2c.r. abd 
+2c.s.a +2c.s.b +a.b.cs +d.s.a +d.s.b +ab.s.cd +r.s.c +ab.c.rs 
+ 10c.s.abr +3c.r.abs +d.r.abs +13d.s.abcr +d.r.abcs +c.s.abdr 
C2: 15rs.cd.ab 

Abbreviated Proof of dSO): T: BS-copy of C over ADR. 
L: -2a. c. +a.b.d +3ad.c.r +a.r.b +a.r.c +a.r.s +bd.s.ac +c.d. 
+c.r.ad +c.s. 

SL: b.t.a +d.t.b +a.b.dt +c.t.abd +b.t.r +a.r.bt +c.t.r +a.r.ct 
+cds.t.abr +bds.t.acr +s.t.a +c.s.t +a.t.cs +bd.s.act +s.t.r +a.r.st 
+bcd.t.ars +r.t.abcds C2L: 4t.bcs.adr 
RS: copy of BD over AC. 

S: b.r.c +a.c.br +4d.r.a +d.r.abc +4b.r.acd +5bd.s.acr +bd.r.acs 
C2: 6rs.bd.ac 

Abbreviated Proof of (ED: T: CDRS-copy of (DR) over 
AB. 

L: -a.b. +3b.dr.a +3a.dr.b +a.b.cr +2a.b.dr +a.b.cs +a.b.ds 
+c.d.r +c.d.s +r.s. 

SL: cr.t.a +cr.t.b +dr.t.a +dr.t.b -t-cs.t.a +cs.t.b +ds.t.a +ds.t.b 
+a.b.crt +a.b.drt +a.b.cst +a.b.dst +c.d.rt +c.d.st +r.s.t 
+abcd.t.rs +abr.t.cds +abs.t.cdr +cr.t.abds +dr.t.abcs +cs.t.abdr 
+ds.t.abcr C2L: 7t.cdrs.ab 
RS: copy of CD over AB. 

S: 2c.r.a +2c.r.b +a.b.cr +5d.r.a +5d.r.b +c.d.r +2ab.r.cd 
+2d.r.abc +8c.r.abd +c.s.a +c.s.b +d.s.a +d.s.b +ab.s.cd 
+ 12cd.s.abr +c.r.abs +d.r.abs +c.rs.abd +cd.r.abs +d.rs.abc C2: 
15rs.cd.ab 

Abbreviated Proof of <|52): T: DS-copy of B over ACR. 
L: -a.b. +a.br.d +3a.b.cr +a.b.cs +a.cr.b +b.c.rs +b.r.ac +c.d. 



+c.s.a +d.s.abc +r.s.c 

SL: a.br.dt +a.bc.st +a.r.bct +ab.t.cd +ad.t.bcrs +2b.t.cr +b.t.rs 
+c.d.t +c.s.brt +cs.t.abdr +d.s.abct +d.t.a +d.t.br +ds.t.abcr 
+r.s.t +r.t.abcds +s.t.a +s.t.bc C2L: 4t.bds.acr 
RS: copy of CD over AB. 

S: a.b.dr +a.b.cs +ab.c.rs +a.c.brs +c.d.r +2cd.r.abs +7d.rs.abc 
+cd.s.abr +2c.rs.abd +5c.s.abr +c.s.a +2c.s.b +ab.r.cd +2d.s.abr 
+d.r.a +2d.r.b +4c.r.a +4c.r.b +d.s.abc +r.s.c C2: 12rs.cd.ab 

Abbreviated Proof of (g3): T: DS-copy of A over BCR. 
L: -a.b. +2a.b.c +c.d. +2a.bc.r +c.r.a +a.r.bc +d.r.bc +a.b.ds 
+r.s.d 

SL: a.t.c +ds.t.a +ds.t.b +c.d.t +b.t.cd +a.t.r +c.r.at +a.b.crt 
+d.t.r +d.r.bct +ds.t.abcr +as.t.bcdr +a.b.dst +cr.t.abds +r.s.dt 
+abc.t.drs C2L: 4t.ads.bcr 
RS: copy of CD over AB. 

S: 5c.r.b +a.b.cr +2d.r.a +2d.r.b +2a.b.dr +c.d.r +ab.r.cd 
+4d.r.abc +4c.r.abd +a.r.bcd +d.s.a +d.s.b +8cd.s.abr +ab.d.rs 
+d.r.abs +2c.s.abdr C2: 10rs.cd.ab 

Abbreviated Proof of (g4): T: CS-copy of B over ADR. 
L: -a.b. +a.br.c +cd.r.b +a.d.b +c.d. +b.r.ad +d.r.b +3a.b.dr 
+d.s.a +a.b.ds +r.s.d +b.d.rs 

SL: c.t.a +a.br.ct +c.d.t +b.t.cd +c.t.br +2d.r.t +2b.t.dr +c.r.bdt 
+ds.t.abcr +as.t.bcdr +s.t.a +a.bd.st +s.t.bd +cr.t.abds +r.s.t 
+b.t.rs +d.s.brt +ac.t.bdrs C2L: 4t.bcs.adr 
RS: copy of CD over AB. 

S: 2c.r.a +3c.r.b +2a.b.cr +4d.r.a +2d.r.b +2c.d.r +2ab.r.cd 
+3d.r.abc +4c.r.abd +a.r.bcd +d.s.b +10cd.s.abr +d.r.abs 
+a.d.brs +2c.rs.abd C2: 12rs.cd.ab 

Abbreviated Proof of (gS): T: DS-copy of C over ABR. 
L: -b.c. +a.b.c +a.b.d +cr.d. +a.r.c +c.r.ab +abd.r.c +3b.c.ar 
+a.s.b +c.s.a +a.s.r +b.r.as +a.c.brs 

SL: d.t.a +d.t.b +a.b.dt +cr.d.t +c.t.abd +2a.r.t +2c.t.ar 
+bd.r.act +a.bs.crt +abs.t.cdr +ac.s.t +s.t.b +bdr.t.acs +s.t.r 
+b.r.st +c.t.brs +s.t.abcdr +d.t.abcrs C2L: 4t.cds.abr 
RS: copy of AD over BC. 

S: 4a.r.b +2a.r.c +d.r.a +d.r.b +a.b.dr +c.d.r +2ab.r.cd 
+8d.rs.abc +c.r.abd +a.s.b +c.s.a +2a.br.cs +a.s.r +b.r.as 
+4a.s.bcr +a.c.brs +4ad.s.bcr C2: 12rs.ad.bc 

Abbreviated Proof of (g6): T: DS-copy of B over ACR. 
L: -b.c. +a.c.b +a.b.d +c.d. +a.r.b +b.r.ac +3b.c.ar +ad.r.b 
+c.r.bd +c.s.a +a.b.cs +a.s.r +b.r.as 

SL: d.t.a +a.b.dt +b.t.cd +c.d.t +2a.r.t +2b.t.ar +d.t.br +d.r.abt 
+c.r.bdt +cs.t.abdr +as.t.bcdr +s.t.ab +c.s.t +b.t.cs +a.c.bst 
+dr.t.abcs +s.t.r +b.r.st +a.s.brt +cd.t.abrs C2L: 4t.bds.acr 
RS: copy of AD over BC. 

S: 2a.r.b +4a.r.c +d.r.a +2d.r.b +a.b.dr +c.d.r +ab.r.cd 
+8d.rs.abc +2c.r.abd +a.r.bcd +2a.s.b +c.s.a +a.br.cs +a.s.r 
+b.r.as +4a.s.bcr +a.r.bcs +2a.c.brs +4ad.s.bcr C2: 12rs.ad.bc 

Abbreviated Proof of lg7]l: T: DR-copy of A over BCS. 
L: -a.b. +2a.bs.c +b.c.a +a.b.d +c.d. +c.r.b +c.s.a +a.s.bc 
+a.b.cs +r.s.c +a.c.rs 

SL: a.t.c +d.t.a +d.t.b +a.b.dt +c.d.t +abrs.t.cd +crs.t.abd +r.t.b 
+r.t.c +c.rs.at +b.c.rt +ads.t.bcr +r.s.t +a.t.rs +bd.t.acrs C2L: 
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4t.adr.bcs 

RS: copy of CD over AB. 

S: c.r.a +a.b.cr +6d.rs.abc +4c.s.b +c.s.abr +4c.r.abs +b.c.ars 
C2: 6rs.cd.ab 

Abbreviated Proof of <g8>: T: CS-copy of B over ADR. 
L: -a.b. +a.b.c +a.d.b +c.d. +r.ac.b +d.r.b +3a.b.dr +b.r.ad 
+d.r.bc +d.s.a +a.b.ds +r.s.d +b.d.rs 

SL: c.ta +c.d.t +b.t.cd +c.t.br +a.br.ct +2d.r.t +2b.t.dr +c.r.bdt 
+s.t.a +s.t.bd +a.bd.st +r.s.t +b.t.rs +d.s.brt +t.ac.bdrs +t.cr.abds 
+t.ds.abcr +t.as.bcdr C2L: 4t.bcs.adr 
RS: copy of CD over AB. 

S: c.r.b +2a.d.r +2d.r.a +2d.r.b +a.r.bcd +4c.r.abd +d.r.abc 
+d.s.b +a.d.brs +d.r.abs +2c.rs.abd +6s.cd.abr C2: 8rs.cd.ab 

■ 

VII. Inequalities Using Three Copy Variables and 
Three Copy Steps 

In this section we will continue the list of inequalities that 
involve three copy variables. The inequalities in this section 
will also involve three copy steps, and hence two lemmas. We 
will give the first one in full detail and then, as before, we 
will quickly turn to abbreviated versions of the proofs. 

Theorem 7: The following is an information inequality 

aI(A;B) < bI(A; B\C) + d(A;C\B) + dI(B;C\A) 
+ e.I(A;B\D) + fI(A;D\B)+gI(B;D\A) 
+ hI(C;D)+U(C;D\A) 
for each of the following values of (a, b, c, d, e, f, g, h, i). 
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(61) 


(3, 


4. 


G, 


4. 


4. 


1. 


0, 


3, 


0) 


(62) 


(4. 


5. 


17. 


G, 


G, 


7. 


0, 


4. 


0) 


(63) 


(4. 


5, 


17, 


13, 


G, 


2, 


0, 


4, 


0) 


(64) 


(3, 


4. 


7, 


5, 


3, 


1, 


0, 


3, 


0) 


(65) 


(G. 


8, 


9, 


9, 


G, 


10, 


1, 


G. 


0) 


(66) 


(G- 


13, 


20, 


2, 


9, 


3. 


0, 


G. 


0) 


(67) 


(4. 


10, 


15, 


1. 


4. 


2- 


2. 


4. 


0) 


(68) 


(4, 


G, 


11. 


3, 


G, 


2- 


0, 


4. 


0) 


(69) 


(3, 


G, 


G. 


1. 


•5. 


4. 


0. 


3. 


0) 


(70) 


(3. 


G, 


8, 


1. 


3. 


2. 


2 


3. 


o) 


(71) 


(4- 


5, 


G. 


G. 


4. 


2. 


2 


4. 


0) 


(72) 


(3. 


8, 


G. 


1. 


3. 


1. 


0. 


3. 


0) 


(73) 


(4. 


14, 


10, 


1. 


G. 


2. 


0. 


4. 


0) 


(74) 


(3, 


4. 


4, 


3, 


3. 


4. 


2- 


3. 


0) 


(75) 


(4, 


13, 


9. 


1. 


7, 


3. 


0, 


4. 


0) 


(76) 


(G, 


8, 


16, 


7, 


G. 


3. 


3, 


G. 


0) 


(77) 



Proof: Proof of d60l : We first show the following 6- 
variable information inequality: 

I(A;B) < 2I(A;BRS\D) + I(A;S) 

+I(A; BR\CS) + I(C; D\S) + I(B; C\RS) 
+I(D;R\AS) + I(A;D\BRS). (78) 



To see this, verify the following 7-variable identity by expand- 
ing into entropies and canceling: 



I(A-B) 

+ I(ARS;T\B) (S) 

+ I(AS;T\D) (S) 

+ I(C;T\AS) (S) 

+ I(S;T\D) (S) 

+ I(C;D\ST) (S) 

+ I(ABR;T\CDS) (S) 

+ I(C;T\RS) (S) 

+ I(A;R\CST) (S) 

+ I{B; C\ARST) (S) 

+ I(D:R\AST) (S) 

+ I(BC; T\ADRS) (S) 

+ I(D; T\ABCRS) (S) 
= 2I(A; BRS\D) + I(A; S) + I(A; BR\CS) 
+ I(C; D\S) + I(B; C\RS) + I(D; R\AS) 
+ I(A;D\BRS) 

+ 3I{T;AC\BDRS) (C2) 

+ (H(TB) - H(AB)) (CI) 

+ 2(H(TD) - H(AD)) (CI) 

- (H(TS) - H{AS)) (CI) 
+ (H(TRS) - H(ARS)) (CI) 

- 3(H(TBDRS) - H(ABDRS)) (CI) 



Each of the terms in the lines marked (S) is a conditional 
mutual information and is therefore nonnegative. Thus, if the 
terms marked (S) are erased and the "=" is replaced by "<", 
then we obtain a 7-variable Shannon-type inequality. By the 
Copy Lemma, we may choose T such that T is a C-copy 
of A over BDRS. Then the term marked (C2) is zero by 
condition (C2), and each of the terms marked (CI) equals 
zero by condition (CI). 

With ( |78| ) verified, we next show the following 5-variable 
information inequality: 



2I(A;B) < 2I(A;BR\C) + 5I{A-BR\D) 

+I(A; DR\B) + 2I(C; D) + 2I(B; C\R) 
+I(D;R,\A) + 2I(A;D\BR). (79) 
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To see this, verify the following 6-variable identity by expand- 
ing into entropies and canceling: 



2I(A;B) 

+ 2I(AR;S\B) (S) 

+ 2I(C;S\A) (S) 

+ I(A;S\D) (S) 

+ I(C;D\S) (S) 

+ 2I(ABR;S\CD) (S) 

+ 2I(C;S\R) (S) 

+ I(A;R\CS) (S) 

+ I{B-C\ARS) (S) 

+ I{BC;S\ADR) (S) 

+ 3J(C; S\ABDR) (S) 

2I(D;S\ABCR) (S) 
[-/(A; B) + 2I(A; BRS\D) + I(A; S) 
+ I(A; BR\CS) + I(C; D\S) + I(B; C\RS) 

+ I{D;R\AS)+I(A;D\BRS)} (LI) 

= 21 (A; BR\C) + 51 (A; BR\D) + I (A; DR\B) 

+ 2I(C; D) + 2I(B; C\R) + I(D; R\A) 

+ 2I(A; D\BR) + 8/(5; AC\BDR) 

+ 8I(S;AC\BDR) (C2) 

-(H(S)-H(A)) (CI) 

+ 2(H(SB) — H(AB)) (CI) 

+ 3(H(SD) - H(AD)) (CI) 

+ (H(SR) - H(AR)) (CI) 

- 5(H(SBDR) - H(ABDR)) (CI) 



The expression in brackets marked with (LI) is nonnegative 
by (|78V Each of the terms in the lines marked (S) is a 
conditional mutual information and is therefore nonnegative. 
Thus, if the terms marked (S) are erased and the "=" is 
replaced by "<", then we obtain a 6-variable Shannon-type 
inequality. By the Copy Lemma, we may choose S such that 
S is a C-copy of A over BDR. Then the term marked (C2) 
is zero by condition (C2), and each of the terms marked (CI) 
equals zero by condition (CI). 

With (l79l l verified, we now continue with the proof of 
(l60l l. By expanding mutual informations into entropies and 
cancelling terms, one can verify the following 5-variable 



identity: 

3I(A; B) 

+ 3I(A;R\B) (S) 

+ I(A;R\C) (S) 

+ I{B;C\AR) (S) 

+ 9I(D;R\B) (S) 

+ I(A;B\DR) (S) 

+ I(C;D\R) (S) 

+ I(AB]R\CD) (S) 

+ 9I(C;R\ABD) (S) 

+ 3I(D;R\ABC) (S) 
+ [-21 (A; B) + 21 (A; BR\C) + 51 (A; BR\D) 
+ I (A; DR\B) + 2I(C; D) + 2I{B; C\R) 

+ I{D;R\A)+2I{A\D\BR)] (L2) 
= AI(A; B\C) + 9I(A; C\B) + 3I{B; C\A) 
+ 6I(A; B\D) + M(A; D\B) + 3J(C; D) 

+ 13I(R;AD\BC) (C2) 

— 3(H(R) — H{A)) (CI) 
+ 12(H(RB) - H(AB)) (CI) 
+ A(H(RC)-H(AC)) (CI) 

- U(H(RBC) - H(ABC)) (CI) 



The expression in brackets "[]" marked (L) is nonnegative by 
Lemma 11.1. Each of the terms in the lines marked (S) is a 
conditional mutual information and is therefore nonnegative. 
Thus, if the terms marked (S) are erased and the "=" is 
replaced by "<", then we obtain a 5-variable Shannon-type 
inequality. By the Copy Lemma, we may choose R such that 
R is a D-copy of C over AB. Then the term marked (C2) is 
zero by condition (C2), and each of the terms marked (CI) 
equals zero by condition (CI). This finishes the proof of (|43| >. 

The expression in brackets marked with (L2) is nonnegative 
by (|79V Each of the terms in the lines marked (S) is a 
conditional mutual information and is therefore nonnegative. 
Thus, if the terms marked (S) are erased and the "=" is 
replaced by "<", then we obtain a 5-variable Shannon-type 
inequality. By the Copy Lemma, we may choose R such that 
R is a D-copy of A over BC. Then the term marked (C2) is 
zero by condition (C2), and each of the terms marked (CI) 
equals zero by condition (CI). 

Abbreviated Proof of d60t : 

We immediately give the abbreviated proof of the same 
inequality, using the same abbreviations as in the last section, 
the main difference being that this time there are two lemmas 
used in the proof. 

T: C-copy of A over BDRS. 
LI: -a.b. +2a.brs.d +a.s. +a.br.cs +c.d.s +b.c.rs +d.r.as +a.d.brs 
SL1: ars.t.b +as.t.d +c.t.as +s.t.d +c.d.st +abr.t.cds +c.t.rs 
+a.r.cst +b.c.arst +d.r.ast +bc.t.adrs +d.t.abcrs C2L1: 3t.ac.bdrs 
S: C-copy of A over BDR. 

L2: -2a.b. +2a.br.c +5a.br.d +a.dr.b +2c.d. +2b.c.r +d.r.a 
+2a.d.br 
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SL2: 2ar.s.b +2c.s.a +a.s.d +c.d.s +2abr.s.cd +2c.s.r +a.r.cs 

+b.c.ars +bc.s.adr +3c.s.abdr +2d.s.abcr 

C2L2: 8s.ac.bdr 

R: D-copy of A over BC. 

S: 3a.r.b +a.r.c +b.c.ar +9d.r.b +a.b.dr +c.d.r +ab.r.cd +9c.r.abd 

+3d.r.abc 

C2: 13r.ad.bc 

Abbreviated Proof of dH): T: BCS-copy of R over AD. 
LI: -a.b. +a.b.c +a.b.d +c.d. +a.d.r +a.r.d +d.r.a 
SL1: b.t.a +c.t.a +c.t.b +a.b.ct +b.t.d +a.d.bt +c.d.t +abrs.t.cd 
+crs.t.abd +drs.t.abc C2L1: 3t.bcrs.ad 
S: BD-copy of R over AC. 

L2: -2a.b. +2a.b.c +2a.b.d +2c.d. +a.c.r +a.r.c +c.r.a +a.d.r 
+a.r.d +d.r.a 

SL2: b.s.a +b.s.c +a.c.bs +d.s.a +d.s.b +a.b.ds +c.d.s +abr.s.cd 

+cr.s.abd +dr.s.abc 

C2L2: 3s.bdr.ac 

R: D-copy of C over AB. 

S: 3c.r.b +3a.b.cr +3d.r.b +3a.b.dr +c.d.r +ab.r.cd +3c.r. abd 

+3d.r.abc 

C2: 7r.cd.ab 

Abbreviated Proof of <|62): T: CS-copy of A over BDR. 
LI: -a.b. +a.br.c +2a.br.d +c.d. +b.c.r +d.r.a +a.d.br 
SL1: ar.t.b +c.t.a +a.t.d +c.d.t +abrs.t.cd +c.t.r +a.r.ct +b.c.art 
+d.r.at +bcs.t.adr +ds.t.abcr C2L1: 3t.acs.bdr 
S: BD-copy of R over AC. 

L2: -2a.b. +a.b.c +a.b.d +2c.d. +a.c.r +a.r.c +c.r.a +b.c.r 
+a.br.c +d.r.a +a.d.br +2a.br.d 

SL2: b.s.a +b.s.c +a.c.bs +d.s.a +d.s.b +a.b.ds +c.d.s +abr.s.cd 

+cr.s.abd +dr.s.abc 

C2L2: 3s.bdr.ac 

R: D-copy of A over BC. 

S: 4a.r.b +3b.c.ar +4d.r.b +a.b.dr +c.d.r +ab.r.cd +4c.r.abd 

+4d.r.abc 

C2: 9r.ad.bc 

Abbreviated Proof of <|63): T: D-copy of C over ABRS. 
LI: -2a. c. +2brs.c.a +2a.b.d +c.d. +2abs.c.r +a.r.c +a.c.br 
+2c.r.bd +c.ds. +a.s.bcd +a.r.cs +as.c.br +d.s.abr +d.r.abcs 
+2d.s.abcr 

SL1: c.t.a +2d.t.a +2a.b.dt +c.d.t +b.t.cd +c.t.abd +c.t.r +a.r.ct 
+b.t.acr +2d.t.br +r.t.abd +2c.r.bdt +2a.t.bcdr +cs.t.a +s.t.d 
+c.d.st +b.t.cds +a.s.bcdt +cs.t.r +a.r.cst +b.c.arst +cd.r.abst 
+c.s.abrt +d.s.abrt +c.t.abdrs +2cd.s.abrt +2d.t.abcrs C2L1: 
6t.cd.abrs 

S: C-copy of A over BDR. 

L2: -3a.b. +3a.br.c +2a.cr.b +2br.c.a +5a.br.d +2a.d.b +3c.d. 

+a.r.c +3b.c.r +d.r.a +5a.d.br +2c.r.bd +d.r.abc 

SL2: 7ar.s.b +c.s.a +a.s.d +c.d.s +abr.s.cd +b.s.cd +dr.s.abc 

+c.s.r +b.c.ars +d.r.as +bc.s.adr +c.s.abdr +3d.s.abcr 

C2L2: 10s.ac.bdr 

R: D-copy of A over BC. 

S: 8a.r.b +b.c.ar +13d.r.b +a.b.dr +c.d.r +ab.r.cd +2a.r.bcd 
+llc.r.abd +7d.r.abc 
C2: 22r.ad.bc 



Abbreviated Proof of |64): T: C-copy of A over BDRS. 
LI: -2a.b. +a.bs.c +2a.br.d +2c.d. +b.c.r +a.br.c +d.r.a +a. ds.br 
+b.c.s +d.s.a +a.dr.bs +2a.brs.d +2c.s.bdr +c.r.abds +c.s.abdr 
SL1: as.t.b +2c.t.a +2a.t.d +2c.d.t +2br.t.cd +ar.t.b +c.t.r +a.r.ct 
+b.c.art +d.r.at +b.t.adr +d.t.abcr +c.t.s +a.s.ct +b.c.ast +d.s.at 
+b.t.ads +d.t.abcs +a.r.bdst +a.s.bdrt +2c.s.bdrt +2a.t.bcdrs 
+ac.r.bdst +ac.s.bdrt +2c.t.abdrs C2L1: 6t.ac.bdrs 
S: BD-copy of A over CR. 

L2: -3a. c. +2a.cr.b +4a.br.d +3c.d. +8a.c.r +a.r.c +2c.r.a 
+2b.c.r +a.r.d +d.r.a +2a.d.br +d.r.abc 

SL2: a.s.c +c.d.s +abr.s.cd +d.s.abc +a.s.r +6bd.s.r +ab.s.r 

+c.r.as +bd.s.acr +d.s.r +a.r.ds +bc.s.adr +d.r.abs +2a.s.bcdr 

+3c.s.abdr 

C2L2: 10s.abd.cr 

R: D-copy of A over BC. 

S: 13a.r.b +llb.c.ar +8d.r.b +2a.b.dr +c.d.r +ab.r.cd +8c.r.abd 
+ 12d.r.abc 
C2: 22r.ad.bc 

Abbreviated Proof of d65>: T: C-copy of A over BDRS. 
LI: -a.c. +a.brs.d +c.d. +cd.r.a +a.bs.dr +ab.c.s +d.s.a +a.dr.bs 
+c.r.abds +c.s.abdr 

SL1: c.t.a +c.d.t +a.t.cd +d.r.t +a.t.dr +c.r.adt +b.t.acdr +a.t.bs 
+c.t.s +a.s.ct +b.c.ast +d.s.at +b.t.ads +d.t.abcs +a.r.bdst 
+ac.r.bdst +ac.s.bdrt +c.t.abdrs C2L1: 3t.ac.bdrs 
S: D-copy of A over BCR. 

L2: -2a.c. +a.cr.b +2br.c.a +a.b.d +2c.d. +2cd.r.a +a.b.cr 

+5a.c.br +a.b.dr +a.d.br +a.br.d +d.r.ab +d.r.abc 

SL2: d.s.b +a.b.ds +c.d.s +a.s.cd +d.s.abc +3a.s.br +c.r.s 

+a.s.cr +c.br.as +3d.s.br +2d.r.abs +d.r.acs +b.s.acdr +3c.s.abdr 

+d.s.abcr 

C2L2: 7s.ad.bcr 

R: D-copy of C over AB. 

S: c.r.a +7c.r.b +d.r.a +3d.r.b +c.d.r +ab.r.cd +2b.r.acd 
+4c.r.abd +8d.r.abc 
C2: 16r.cd.ab 

Abbreviated Proof of (|66): T: D-copy of A over BCRS. 
LI: -3a. c. +a.crs.b +bs.c.a +a.b.d +c.d. +2a.bs.cr +5a.c.br 
+2c.br.a +2a.b.dr +d.r.ab +2c.dr. +a.br.cs +c.s.ad +3ad.s.bcr 
+2d.s.bcr +d.r.abcs 

SL1: d.t.b +a.b.dt +c.d.t +a.t.cd +3a.t.br +2c.r.t +2a.t.cr 
+2b.c.art +2d.t.ar +2d.t.br +2a.b.drt +d.r.abt +2c.dr.t +2b.t.cdr 
+3c.t.abdr +s.t.ab +c.s.t +a.t.cs +b.c.ast +d.t.as +c.s.adt 
+b.t.acds +a.r.bcst +2a.s.bcrt +2d.s.bcrt +2a.t.bcdrs +r.ad.bcst 
+3s.ad.bcrt +3d.t.abcrs C2L1: 9t.ad.bcrs 
S: C-copy of A over BDR. 

L2: -4a. c. +a.cr.b +3br.c.a +a.b.d +2c.d. +2a.b.cr +6a.c.br 
+4a.b.dr +10a.d.br +a.br.d +b.d.ar +d.r.ab +2c.dr. +c.r.ad 
+d.r.ac +d.r.abc 

SL2: c.d.s +a.s.cd +d.s.abc +2a.s.br +c.s.ar +9c.s.br +c.r.abs 

+2a.s.dr +b.d.ars +c.r.ads +d.r.acs +2a.s.bcdr +2b.s.acdr 

+c.s.abdr +4d.s.abcr 

C2L2: 13s.ac.bdr 

R: D-copy of C over AB. 

S: 3c.r.a +9c.r.b +6d.r.a +15d.r.b +2c.d.r +2b.r.acd +4r.ab.cd 
+ 17c.r.abd +lld.r.abc 
C2: 35r.cd.ab 
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Abbreviated Proof of <|67]l: T: C-copy of A over BDRS. 
LI: -3a. d. +2a.b.c +3a.brs.d +3c.d. +a.brs.c +d.r.a +c.ds.r 
+2a.bdr.s +2d.s.a +3a.dr.bs +2d.s.bc +a.bd.rs +a.b.drs 
SL1: 3c.t.a +2a.b.ct +3a.t.d +3c.d.t +2b.t.cd +c.t.r +a.r.ct +d.r.at 
+2a.t.s +2c.t.bs +2d.s.at +2d.s.bct +a.t.rs +c.s.art +t.cr.bs 
+a.b.crst +r.t.ads +a.b.drst +c.d.rst +t.ab.cdrs +c.t.abdrs 
+d.t.abcrs +t.bcr.ads +2t.ar.bcds +t.abrs.cd +2t.drs.abc 
+t.bcs.adr C2L1: 10t.ac.bdrs 
S: D-copy of C over ABR. 

L2: -4a.c. +4a.br.c +a.cr.b +a.b.d +3c.d. +a.r.c +lla.c.br 
+2b.c.ar +2d.r.a +a.b.dr +3a.d.br +6a.br.d +d.r.ab +c.d.r +d.cr. 
+2d.r.bc 

SL2: d.s.b +a.b.ds +2a.r.s +c.s.ar +4c.s.br +a.b.crs +r.ad.s 

+10d.s.br +d.r.abs +d.cr.s +2d.r.bcs +2a.s.bcdr +2s.ab.cdr 

+llc.s.abdr +2d.s.abcr 

C2L2: 17s.cd.abr 

R: D-copy of A over BC. 

S: 17a.r.b +a.r.c +lld.r.b +a.b.dr +c.d.r +2a.r.bcd +3r.ab.cd 
+12c.r.abd +18d.r.abc 
C2: 35r.ad.bc 

Abbreviated Proof of <|68): T: C-copy of A over BDRS. 
LI: -2a. b. +2a.b.c +2b.d.a +2c.d. +d.ar.b +d.r.ac +a.b.drs 
+2a.brs.d +2a.rs.bd +a.bs.dr +d.ars.b +d.rs.ac 
SL1: 2c.t.b +2a.b.ct +2c.d.t +2a.t.cd +t.ar.b +c.t.ar +d.r.t 
+a.t.dr +b.d.art +d.r.act +t.ars.b +c.t.ars +d.rs.t +a.t.drs 
+b.d.arst +d.rs.act +b.t.acdrs +c.t.abdrs +2t.drs.abc H-t.cs.abdr 
+t.bs.acdr C2L1: 6t.ac.bdrs 
S: D-copy of A over BCR. 

L2: -3a.b. +2a.c.b +2c.d. +5a.cr.b +5a.br.c +b.c.ar +a.b.dr 

+2a.dr.b +3a.br.d +b.d.ar +d.br.a +d.cr. +2d.r.ac 

SL2: 2s.ar.b +r.s.c +2s.ar.c +b.c.ars +d.s.ar +6s.dr.b +c.d.rs 

+b.s.acdr H-s.ab.cdr +7c.s.abdr +d.s.abcr 

C2L2: 10s.ad.bcr 

R: D-copy of A over BC. 

S: 3a.r.b +3a.r.c +2d.r.a +6d.r.b +c.d.r +2b.r.acd +2r.ab.cd 
+8c.r.abd +lld.r.abc 
C2: 23r.ad.bc 

Abbreviated Proof of d69): T: C-copy of A over BDRS. 
LI: -2a.c. +2c.d. +a.c.br +2d.r.a +a.c.bs +r.s.abc +s.dr.a 
+a.b.drs +2a.bd.rs +4a.brs.d +c.ds.r 

SL1: 2c.t.a +a.b.ct +2a.t.d +2c.d.t +c.t.r +a.r.ct +c.t.br 
+d.r.at +c.t.bs +r.s.t +2a.t.rs +c.s.art +r.s.bct +a.b.crst +d.rs.at 
+a.b.drst +c.d.rst +t.ab.cdrs +c.t.abdrs +d.t.abcrs +2t.abrs.cd 
+t.drs.abc +t.bcs.adr C2L1: 7t.ac.bdrs 
S: D-copy of C over ABR. 

L2: -2a.b. +2a.b.c +2c.d. +c.r.a +c.r.b +2a.b.cr +2c.r.ab +c.br.a 

+5c.ar.b +2d.r.a +a.b.dr +2a.d.br +4a.br.d +c.d.r 

SL2: c.s.b +r.s.a +7s.dr.b H-s.ab.cdr +7c.s.abdr +d.s.abcr 

C2L2: 9s.cd.abr 

R: D-copy of C over AB. 

S: era +c.r.b +8d.r.b +a.b.dr +c.d.r +2r.ab.cd +8c.r.abd 
+ 10d.r.abc 
C2: 20r.cd.ab 



Abbreviated Proof of <70): T: D-copy of A over BCRS. 
LI: -a.b. +c.d. +a.b.cs +b.as.d +c.rs.a +a.brs.c +a.bcs.r +d.r.bcs 
+d.r.abcs 

SL1: d.t.b +c.d.t +c.s.t +a.t.cs +d.t.as +b.as.dt +t.bs.cd 
+c.t.abds +t.as.r +c.r.ast +a.b.crst +d.r.bcst +a.t.bcdrs 
+r.ad.bcst +d.t.abcrs C2L1: 3t.ad.bcrs 
S: C-copy of A over BDR. 

L2: -2a.c. +a.c.b +a.b.d +2c.d. +c.r.a +a.b.cr +2a.bc.r +d.r.a 

+a.b.dr +4a.d.br +3a.br.d +c.d.r +d.r.bc +d.r.abc 

SL2: c.s.a +2a.s.d +c.d.s +b.s.cd +c.s.abd +2a.s.r +4c.s.br 

+a.b.crs +d.r.as +a.b.drs +c.d.rs +a.s.bcdr H-s.ab.cdr +c.s.abdr 

+2d.s.abcr +2s.dr.abc 

C2L2: 8s.ac.bdr 

R: D-copy of C over AB. 

S: 4c.r.b +8d.r.b +a.r.bcd +r.ab.cd +8c.r.abd +2d.r.abc 
C2: 13r.cd.ab 

Abbreviated Proof of (ED: T: D-copy of B over ACRS. 
LI: -a.brs. +a.rs. +c.s.br +2a.b.crs +b.c.ars +c. as.br +a.b.drs 
+c.d.rs 

SL1: b.t.ars +c.s.rt +b.t.crs +a.c.brst +d.t.ars +d.t.brs +a.b.drst 
+c.ds.rt -H.ab.cdrs +c.t.abdrs +d.t.abcrs C2L1: 3t.bd.acrs 
S: C-copy of D over ABR. 

L2: -2a. d. +2a.b.cr +a.c.br +b.c.ar +a.b.dr +4a.d.br +b.d.ar 
+d.br.a +2d.cr. 

SL2: 3s.cr.a +c.s.br +2d.s.ar +2d.s.br +a.b.drs +c.d.rs 

H-2s.ab.cdr +2c.s.abdr +4d.s.abcr 

C2L2: 8s.cd.abr 

R: D-copy of A over BC. 

S: 8a.r.b +3a.r.c +2d.r.a +5d.r.b +c.d.r +3r.ab.cd +7c.r.abd 

+4d.r.abc 

C2: 14r.ad.be 

Abbreviated Proof of (72) : T: D-copy of C over ABRS. 
LI: -2a.b. +2a.b.c +a.b.cr +a.b.dr +c.dr. +a.b.crs +2c.rs.ab 
+2c.brs.a +2c.ars.b +a.b.drs +c.drs. 

SL1: t.cr.a +t.cr.b +a.b.crt +t.dr.a +t.dr.b +a.b.drt +c.d.rt 
+t.crs.a +t.crs.b +a.b.crst +t.drs.a +t.drs.b +a.b.drst +c.d.rst 
+t.ab.cdrs +c.t.abdrs +d.t.abcrs +t.ds.abcr +t.cs.abdr H-t.abs.cdr 
C2L1: 6t.cd.abrs 
S: C-copy of D over ABR. 

L2: -3a.b. +3a.b.c +a.b.cr +a.c.br +b.c.ar +3c.r.ab +c.br.a 

+c.ar.b +d.r.a +d.r.b +4a.b.dr +2d.br.a +2d.ar.b +3c.dr. 

SL2: s.cr.a +s.cr.b +2s.dr.a +2s.dr.b +a.b.drs +c.d.rs H-2s.ab.cdr 

+2c.s.abdr +7d.s.abcr 

C2L2: lls.cd.abr 

R: D-copy of C over AB. 

S: 2c.r.a +2c.r.b +3d.r.a +3d.r.b +c.d.r +4r.ab.cd +8c.r.abd 

+8d.r.abc 

C2: 20r.cd.ab 

Abbreviated Proof of (|73): T: BC-copy of S over ADR. 
LI: -a.s. +a.r. +a.b.cr +a.d.br +c.d.r +a.r.s +2a.s.dr +d.s.ar 
SL1: 2a.r.t +b.t.ar +c.t.ar H-ct.br +a.b.crt +b.t.dr +a.d.brt 
+c.d.rt +t.ds.abcr +t.cs.abdr H-t.abs.cdr C2L1: 3t.bcs.adr 
S: D-copy of C over ABR. 

L2: -2a.c. +2a.r. +a.b.cr +4a.c.br +a.br.c +b.c.ar +3a.b.dr 
+a.d.br +2c.d.r 
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SL2: a.r.s +c.s.ar +c.s.br +a.b.crs +3d.s.br +3a.b.drs +c.d.rs 

H-s.ab.cdr +3c.s.abdr +d.s.abcr 

C2L2: 5s.cd.abr 

R: D-copy of A over BC. 

S: 6a.r.b +2a.r.c +3d.r.a +4d.r.b +c.d.r +3r.ab.cd +4c.r.abd 

+7d.r.abc 

C2: 14r.ad.bc 

Abbreviated Proof of <|74|: T: BCR-copy of S over AD. 
LI: 2(-a.b. +a.b.c +a.b.d +c.d. +a.d.s +a.s.d +d.s.a) 
SL1: b.t.a +c.t.a +c.t.b +a.b.ct +b.t.d +a.d.bt +c.d.t +t.drs.abc 
+t.crs.abd +t.abrs.cd C2L1: 3t.bcrs.ad 
S: BD-copy of C over AR. 

L2: -3a.c. +2a.b.c +2a.d.b +3c.d. +6a.c.r +a.r.c +c.r.a +5a.r.d 
+d.r.a 

SL2: c.s.a +c.d.s +c.s.r +a.r.cs +5d.s.r +5a.r.ds +d.r.as 

+6s.bc.adr +s.bd.acr +s.abr.cd 

C2L2: 8s.bcd.ar 

R: D-copy of C over AB. 

S: 8c.r.b +8a.b.cr +6d.r.b +6a.b.dr +c.d.r +r.ab.cd +6c.r. abd 

+8d.r.abc 

C2: 15r.cd.ab 

Abbreviated Proof of d75): T: D-copy of A over BCRS. 
LI: -a.c. +c.d. +r.cd.a +c.s.a +b.d.s +a.bs.d +2a.cr.bs +a.bs.cr 
+d.r.abcs +d.s.abcr 

SL1: d.t.a +c.d.t +a.t.cd +c.r.t +a.t.cr +d.r.act +b.t.acdr +a.t.bs 
+c.s.at +b.t.acs +d.t.s +a.s.dt +b.d.ast +c.t.abds +a.r.bcst 
+r.ad.bcst +s.ad.bcrt +d.t.abcrs C2L1: 3t.ad.bcrs 
S: C-copy of A over BDR. 

L2: -2a.c. +2c.d. +a.b.cr +2a.c.br +c.ar.b +a.b.dr +4a.d.br 
+2a.br.d +2d.br.a +2r.cd.a +d.r.abc 

SL2: c.s.b +a.b.cs +c.d.s +a.s.cd +d.s.abc +3a.s.br +3c.s.br 

+c.r.abs +d.r.s +a.s.dr +d.r.abs +d.br.as +c.r.ads +b.s.acdr 

+2c.s.abdr +2d.s.abcr 

C2L2: 7s.ac.bdr 

R: D-copy of C over AB. 

S: c.r.a +3c.r.b +d.r.a +7d.r.b +c.d.r +2b.r.acd +r.ab.cd 
+9c.r.abd +3d.r.abc 
C2: 16r.cd.ab 

Abbreviated Proof of d76): T: BDR-copy of S over AC. 
LI: 2(-a.b. +a.b.c +a.b.d +c.d. +a.c.s +a.s.c +c.s.a) 
SL1: b.t.a +b.t.c +a.c.bt +d.t.a +d.t.b +a.b.dt +c.d.t +t.drs.abc 
+t.crs.abd +t.abrs.cd C2L1: 3t.bdrs.ac 
S: BC-copy of D over AR. 

L2: -3a.c. +2a.b.c +2a.d.b +3c.d. +5a.c.r +c.r.a +a.d.r +6a.r.d 
+d.r.a 

SL2: d.s.a +c.d.s +5c.s.r +5a.r.cs +c.r.as +d.s.r +a.r.ds 

+s.bc.adr +6s.bd.acr +s.abr.cd 

C2L2: 8s.bcd.ar 

R: D-copy of C over AB. 

S: 6c.r.b +6a.b.cr +8d.r.b +8a.b.dr +c.d.r +r.ab.cd +8c.r.abd 

+6d.r.abc 

C2: 15r.cd.ab 

Abbreviated Proof of d77>: T: C-copy of A over BDRS. 
LI: -3a. d. +a.b.c +c.d. +2a.b.cr +c.r.ab +a.d.br +a.dr.b +2d.br.a 



+2d.cr. +d.bs.a +d.s.ac +a.br.ds +4a.ds.br +2a.bs.dr +2c.s.bdr 
+c.r.abds +3c.s.abdr 

SL1: c.t.b +a.b.ct +c.d.t +a.t.cd +3a.t.br +2c.t.ar +2c.t.br 
+2a.b.crt +c.r.abt +2d.r.t +2a.t.dr +2b.d.art +2d.cr.t +2b.t.cdr 
+3d.t.abcr +s.t.ab +c.t.as +d.s.t +a.t.ds +b.d.ast +d.s.act 
+b.t.acds +a.r.bdst +2a.s.bdrt +2c.s.bdrt +2a.t.bcdrs +r.ac.bdst 
+3s.ac.bdrt +3c.t.abdrs C2L1: 9t.ac.bdrs 
S: D-copy of A over BCR. 

L2: -4a. c. +b.d.a +2c.d. +2a.b.cr +13a.c.br +c.br.a +c.ar.b 
+d.r.a +4a.b.dr +3a.d.br +2a.br.d +2d.br.a +2c.dr. +2d.r.ac 
+r.cd.ab 

SL2: c.d.s +a.s.cd +c.s.abd +2a.s.br +2c.r.s +2a.s.cr +b.c.ars 

+d.s.ar +9d.s.br +d.r.abs +c.r.ads +d.r.acs +2a.s.bcdr +2b.s.acdr 

+4c.s.abdr +d.s.abcr 

C2L2: 13s.ad.bcr 

R: D-copy of C over AB. 

S: 4c.r.a +15c.r.b +3d.r.a +9d.r.b +2c.d.r +2b.r.acd +4r.ab.cd 
+ llc.r.abd +17d.r.abc 
C2: 35r.cd.ab 



VIII. Inequalities Using Four Copy Variables with 
at most Three Copy Steps 

In this section we list the inequalities that can be derived 
from this method using at most four auxiliary variables and 
at most three instances of the Copy Lemma. Therefore, for 
the sake of completeness, this list includes those found in 
previous sections. Each entry in the following list begins with 
a list of nine integers. These integers represent the coefficients 
a, b, c, d, e, /, g, h, i for the non-Shannon inequality of the 
form 

al(A-B) < bI(A;B\C) + d(A;C\B)+dI(B;C\A) 
+ eI{A- B\D) + fI{A; D\B) + gI(B; D\A) 
+ hI(C;D)+U(C;D\A) 

Following the list of integers, is a list of copy steps used in the 
proof of the inequality. Each copy step is ended by a period. 

1) 242131020rc ab.s r ad. 

2) 2 3 3 1 5 2 2 rs cd ab.t r ad. 

3) 3 6 3 1 6 3 3 rs cd ab.t r ad. 

4) 2 4 2 1 2 2 3 rs bd act c adr. 

5) 2 3 3 2 2 2 rs cd ab.t b acr. 

6) 4 6 4 3 4 2 1 4 rs cd ab.t b adr. 

7) 2 5 2 1 2 2 rs ad bet c abr. 

8) 2 4 3 1 2 2 rs ad bet b acr. 

9) 2 4 1 2 2 3 2 rs cd ab.t a bcs. 

10) 374141030rc ab.s r act r ad. 

1 1) 4 6 1 1 3 6 2 4 r c ab.s c abr.t a bdrs. 

12) 3 6 3 1 4 1 3 5 rs bd ac.tu cs adr. 

13) 7 8 12 12 7 5 5 7 rs cd ab.tu (cr)(cs) ab. 

14) 5 14 9 1 7 2 5 rs cd ab.tu cd ar. 

15) 6 7 11 11 6 3 3 6 rs cd ab.tu (cr)(dr) ab. 

16) 346362030rc ab.st cd abr.u (cr) at. 

17) 11 23 28 3 11 7 5 11 r a best cd abr.u b acrt. 

18) 568753250rc ab.st cd abr.u b acrt. 

19) 6 12 6364360rb ac.st cd ab.u b act. 
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20) 4 5 16 4 10 6 4 r a best ca bdr.u (at) bdr. 78 

21) 365153030rc ab.st ac bdr.u r ads. 79 

22) 4 13 7 1 4 2 2 4 r b ac.st cd ab.u a bcrs. 80 

23) 457641140rc ab.st cd ab.u b acrs. 81 

24) 4 8 4 1 10 6 4 r c ab.st be ad.u r as. 82 

25) 5 16 13 1 5 1 1 5 r a best cd abr.u s acr. 83 

26) 5 6 1 1 1 1 5 1 1 5 r c ab.st cd ab.u (cs) abr. 84 

27) 234145020 rscd ab.t c abrs.u a dst. 85 

28) 4 5 6 4 4 2 4 4 rs cd ab.t d abrs.u b acrt. 86 

29) 4 7 4 2 4 1 1 4 7 rs bd act c adr.u a rst. 87 

30) 2 3 3 1 2 3 2 2 rs cd ab.t d abrs.u a bcrt. 88 

31) 4 10 2 3 9 9 4 rs cd ab.t c as.u r bdst. 89 

32) 4 7 3 4 5 5 4 rs cd ab.t a bdr.u c abrst. 90 

33) 3 5 4 2 3 3 4 rs bd act c abs.u b acrst. 91 

34) 5 14 1 1 1 5 2 2 5 rs cd ab.t a bcr.u a bcrt. 92 

35) 6 15 10 2 6 6 11 rs bd act c adr.u a bcrt. 93 

36) 11 31 18 3 13 4 11 rs cd ab.t c ar.u d art. 94 

37) 18 38 31 6 18 6 6 18 rs cd ab.t a bcr.u b acrt. 95 

38) 4 9 3 2 8 4 4 rs cd ab.t c as.u r bst. 96 

39) 5 12 3 3 10 9 5 rs cd ab.t c as.u r bdst. 97 

40) 8 19 6 4 9 14 8 rs cd ab.t a bdr.u a bdrt. 98 

41) 3 5 4 2 4 1 3 rs cd ab.t d ar.u b acrt. 99 

42) 7 19 8 1 9 8 2 7 rs cd ab.t d ar.u c art. 100 

43) 6 16 2 9 6 11 6 rs cd ab.t a bdr.u b acrt. 101 

44) 7 8 11 11 7 7 7 7 rs cd ab.t c abrs.u (dr) abt. 102 

45) 5 8 10 3 5 1 1 5 rs cd ab.t c ar.u b acrt. 103 

46) 4 10 10 1 4 1 4 4 rs cd ab.t a bcr.u b adrt. 104 

47) 8 9 14 14 8 8 8 8 rs cd ab.t c abrs.u (dr) abt. 105 

48) 3 5 4 1 8 5 3 rs cd ab.t c as.u a bdst. 106 

49) 6 11 10 2 6 6 9 6 rs cd ab.t d abr.u a bcrt. 107 

50) 7 19 11 2 7 5 10 7 rs cd ab.t b adr.u a bcrt. 108 

51) 6 13 10 2 6 2 2 6 rs cd ab.t a bcr.u b acrt. 109 

52) 9 12 16 7 9 13 11 9 rs cd ab.t (dr) ab.u a bcrt. 110 

53) 7 8 16 16 7 3 3 7 rs cd ab.t (cr) ab.u c abrt. Ill 

54) 5 9 4 4 5 3 1 5 rs cd ab.t a bcs.u (cs) abrt. 112 

55) 8 17 7 3 16 8 8 rs cd ab.t c as.u c bst. 113 

56) 3 9 2 2 3 3 rs bd act b ar.u c abrt. 1 14 

57) 9 17 15 3 9 15 5 9 rs cd ab.t d abrs.u a bcrt. 115 

58) 3 4 5 4 4 1 3 rs cd ab.t a bcr.u a bdst. 1 16 

59) 7 16 12 2 7 4 3 7 rs cd ab.t a bcr.u b adst. 117 

60) 3 5 6 2 3 3 rs ad bet a cr.u b acrt. 118 

61) 6 11 10 2 6 9 4 6 rs cd ab.t d abrs.u a bcrt. 119 

62) 6 7 13 13 6 2 2 6 rs cd ab.t (cr) ab.u c abrst. 120 

63) 10 23 16 3 10 5 5 10 rs cd ab.t a bcr.u d abst. 121 

64) 4 5 9 6 6 3 4 rs cd ab.t a bcr.u a bdst. 122 

65) 9 19 8 4 15 7 9 rs cd ab.t c as.u s act. 123 

66) 10 11 22 22 10 9 9 10 rs cd ab.t (cr) ab.u (dr) abt. 124 

67) 5 20 15 1 5 5 rs ad bet (ar) bcu (ar) bet. 125 

68) 4 8 11 1 4 4 4 4 rs cd ab.t a bcr.u s abedt. 126 

69) 7 16 12 2 7 3 4 7 rs cd ab.t a bcr.u a bdst. 127 

70) 4 9 8 1 4 3 2 4 rs cd ab.t a bcr.u b adst. 128 

71) 6 12 5 5 6 3 6 rs cd ab.t a bcs.u c abrt. 129 

72) 7 12 10 3 14 6 7 rs cd ab.t c as.u r act. 130 

73) 13 24 16 5 26 10 13 rs cd ab.t c as.u s act. 131 

74) 6 14 4 3 12 11 6 rs cd ab.t c as.u c bdt. 132 

75) 10 18 11 7 13 3 10 rs cd ab.t a bdr.u r abedt. 133 

76) 12 22 14 5 23 9 12 rs cd ab.t r ad.u a bdrt. 134 

77) 4 14 9 1 4 4 rs ad bet (ar) bcu c abrt. 135 



6 1 1 10 2 6 8 5 6 rs cd ab.t d abrs.u a bcrt. 
5 11 4 2 10 8 5 rs cd ab.t c as.u c bdt. 
8 14 13 4 12 8 8 rs cd ab.t a bcs.u a bdrt. 
10 20 9 6 12 7 10 rs cd ab.t a bcs.u c abrt. 
8 17 9 5 8 8 rs ad bet c abr.u c abrt. 

3 8 1 4 6 7 3 rs cd ab.t c as.u t cs. 

8 9 15 15 8 6 6 8 rs cd ab.t (cr) ab.u (dr) abt. 
78212172270 rscd ab.t (cr) ab.u (cr) abt. 
5 6 15 15 5 5 rs cd ab.t (cr) ab.u (cr) abt. 

5 6 5 5 5 10 10 5 rs cd ab.t (cs) ab.u (ds) abt. 

9 10 19 19 9 7 7 9 rs cd ab.t (cr) ab.u (dr) abt. 

6 7 17 17 6 1 1 6 rs cd ab.t (cr) ab.u (cr) abt. 

10 17 11 11 10 10 rs cd ab.t b acr.u b acrt. 

4 9 3 2 5 7 4 rs cd ab.t a bcs.u (at) bcs. 
4 6 5 4 4 4 4 rs cd ab.t a bcs.u (bt) acr. 

4 5 10 9 4 4 rs cd ab.t (cr) ab.u b acrt. 
3 4 5 5 3 3 rs cd ab.t b acr.u t bcr. 

6 13 5 3 8 2 6 10 rs bd act c adr.u t rs. 

3 4 4 5 4 1 3 rs cd ab.t a bcr.u t acs. 

7 25 18 1 13 6 7 rs cd ab.t c ar.u t ad. 

10 17 8 9 10 4 2 10 rs cd ab.t a bcs.u a best. 
9 16 14 4 15 9 9 rs cd ab.t a bcs.u (rt) ad. 

9 10 18 18 9 8 8 9 rs cd ab.t (cr) ab.u (cs) abt. 
7 15 6 3 12 7 7 rs cd ab.t c as.u t cs. 
365131330 rscd ab.t c abrs.u t adr. 

5 9 6 2 10 5 5 rs cd ab.t c as.u t abc 
7 35 28 1 1 1 4 7 rs cd ab.t r aeu s at. 

4 7 5 1 13 8 4 rs cd ab.t c as.u t ad. 

7 19 2 11 7 12 1 7 rs cd ab.t a bcs.u b acrt. 
4 13 9 1 4 4 7 rs bd act c abs.u t ar. 

4 13 10 1 4 4 rs ad bet (ar) bcu b acrt. 

3 8 5 1 3 3 rs ad bet b acr.u t abr. 
362240135 rsbd act c adr.u t rs. 

4 9 8 1 4 2 3 4 rs cd ab.t a bcr.u a bdst. 

5 7 1 1 6 5 2 5 rs cd ab.t (cr) ab.u t adr. 

5 10 5 1 15 10 5 rs cd ab.t c as.u t ad. 
4 11 7 1 4 4 5 4 rs cd ab.t b acs.u t ar. 

8 17 7 4 10 2 8 12 rs bd act c adr.u t rs. 

7 8 9 9 7 14 14 7 rs cd ab.t (cs) ab.u (ds) abt. 
4 5 10 6 7 2 4 rs cd ab.t (cr) ab.u (rt) ad. 

10 26 3 16 10 18 1 10 rs cd ab.t a bdr.u b acrt. 

9 12 22 10 13 10 9 rs cd ab.t (cr) ab.u a bdst. 

3 7 2 2 3 3 2 rs bd act c abs.u t bcr. 

7 9 16 10 10 2 7 rs cd ab.t (cr) ab.u (rt) ad. 

6 9 6 4 6 3 4 6 rs cd ab.t b acs.u t bcr. 

4 8 2 5 4 4 4 rs bd act c abs.u (at) bcr. 
13 37 22 3 17 6 13 rs cd ab.t c ar.u d art. 

8 10 23 12 8 1 4 8 rs cd ab.t (cr) ab.u a best. 
6777611 1160 rscd ab.t (cs) ab.u (ds) abt. 

11 29 3 19 11 21 2 11 rs cd ab.t a bcs.u b acrt. 

9 12 8 13 9 9 2 9 rs cd ab.t a bcs.u a bcrt. 
4 7 5 2 8 2 4 rs cd ab.t r ad.u r adt. 

4 1 1 7 1 4 1 7 4 rs cd ab.t b acs.u a bcrt. 
8 14 6 7 8 4 2 8 rs cd ab.t a bcs.u c abrst. 

6 8 6 5 6 4 5 6 rs cd ab.t b adr.u t bdr. 

7 13 5 6 7 5 1 7 rs cd ab.t a bdr.u r abedt. 

12 16 25 18 12 3 12 rs cd ab.t (cr) ab.u t acs. 

8 12 9 5 8 4 8 8 rs cd ab.t b acs.u t bcr. 
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5 6 14 9 5 1 2 5 rs cd ab.t (cr) ab.u a bdrt. 

15 27 45 5 15 27 11 15 rs cd ab.t (cs) ab.u a bcrt. 

10 18 12 4 21 7 10 rs cd ab.t r ad.u r adt. 
9 17 9 6 11 4 9 rs cd ab.t a bcs.u c abrst. 

11 29 16 3 15 6 11 rs cd ab.t s ac.u r act. 

18 22 51 30 18 3 6 18 rs cd ab.t (cr) ab.u a best. 

3 6 6 1 3 1 1 3 rs ad bet b ars.u b acr. 

18 32 13 16 18 13 2 18 rs cd ab.t a bcs.u (bt) acs. 

6 14 9 2 6 2 3 6 rs cd ab.t a bcr.u (ds) abt. 

14 24 13 14 14 11 14 rs cd ab.t a bcs.u (bt) acr. 
595462050 rscd ab.t a bcs.u c abrt. 

6 17 2 9 9 3 6 rs cd ab.t c br.u t ad. 

4 1 1 7 1 4 2 6 4 rs cd ab.t b acs.u t ar. 

18 30 20 12 37 15 18 rs cd ab.t c as.u b acrt. 

8 10 22 13 8 1 2 8 rs cd ab.t (cr) ab.u a bdrt. 
583555250 rscd ab.t a bcs.u t acr. 

9 19 7 5 12 8 9 rs cd ab.t a bcs.u t as. 
454446540 rscd ab.t (dr) ab.u b adst. 

14 28 10 13 14 14 11 rs bd act c abs.u (at) bcr. 

7 15 6 4 8 6 7 rs cd ab.t a bdr.u r abedt. 

7 18 26 2 7 2 1 7 rs da bet (as) bc.u a bdst. 

11 19 34 4 11 20 8 11 rs cd ab.t (cs) ab.u a bcrt. 

4 5 5 4 4 3 3 4 rs cd ab.t b acr.u (dr) ab. 

5 13 8 1 8 3 5 rs cd ab.t c ar.u r ad. 

6 7 15 12 6 2 3 6 rs cd ab.t (cr) ab.u a best. 
14 21 17 9 14 6 12 14 rs cd ab.t b acs.u t bcr. 
352432030 rscd ab.t a bdr.u t ar. 
474362040 rscd ab.t d ar.u a bcr. 

19 32 21 12 40 17 19 rs cd ab.t c as.u b acrt. 

10 20 9 7 11 5 10 rs cd ab.t a bcs.u c abrt. 

8 21 34 2 8 3 2 8 rs ad bet (ar) bc.u r abst. 
686762160 rscd ab.t b adr.u a bcr. 

4 10 2 3 5 1 4 4 rs bd act b adr.u c abr. 
4 5 9 7 5 1 4 rs ad bet a bdr.u a cr. 
585354450 rscd ab.t b acs.u t bcr. 

4 13 1 9 4 7 4 rs cd ab.t a bdr.u t be. 

9 25 12 3 1 1 4 9 rs cd ab.t d ar.u c art. 
567654450 rscd ab.t b acr.u (dr) ab. 
699582060 rscd ab.t b acr.u r ad. 

7 1 1 6 6 7 3 2 7 rs cd ab.t a bdr.u b adr. 
495173040 rscd ab.t c ar.u r ad. 

9 18 7 6 12 9 9 rs cd ab.t a bdr.u t ad. 

3 7 2 2 3 4 3 rs cd ab.t a bdr.u r abedt. 

7 14 6 4 10 7 7 rs cd ab.t a bcs.u t as. 

688562160 rscd ab.t b acr.u b acs. 

9 24 31 2 9 12 3 9 rs ad bet a bdr.u (ar) bet. 

9 24 43 2 9 4 3 9 rs ad bet (ar) bc.u r abst. 

597253250 rscd ab.t a bcr.u b acs. 

13 20 8 14 13 12 6 13 rs cd ab.t a bdr.u t acr. 

57745 1250 rscd ab.t b acr.u b adr. 

21 32 24 13 21 10 18 21 rs cd ab.t b acs.u t bcr. 

5 10 3 6 5 5 2 rs bd act c adr.u a bcr. 

6 1 1 6 3 12 6 6 rs cd ab.t a bcr.u r ad. 

20 34 17 18 20 5 4 20 rs cd ab.t a bcs.u a best. 
24 40 17 24 24 13 6 24 rs cd ab.t a bdr.u d abst. 
5 1 1 14 1 5 6 6 5 rs cd ab.t a bcr.u (ds) abt. 

7 18 2 21 7 1 7 7 3 rs bd act b adr.u (br) act. 

8 10 14 13 8 2 1 8 rs cd ab.t b acs.u (cr) ab. 



194 
195 
196 
197 
198 
199 
200 
201 
202 
203 
204 
205 
206 
207 
208 
209 
210 
211 
212 
213 
214 



5 8 6 5 5 2 5 rs cd ab.t b acr.u a bdr. 

19 29 22 12 19 8 16 19 rs cd ab.t b acs.u t bcr. 

4 6 5 4 6 2 4 rs cd ab.t a bcr.u r ad. 

26 43 25 21 26 6 7 26 rs cd ab.t b acs.u b acst. 
11 19 28 4 11 20 12 11 rs cd ab.t (cs) ab.u a bcrt. 

5 10 19 1 5 8 8 5 rs cd ab.t a bcr.u (ds) abt. 

7 13 6 7 7 3 7 rs cd ab.t a bdr.u r abedt. 

36 61 35 29 36 6 11 36 rs cd ab.t b acs.u b acst. 

6 9 8 4 6 2 5 6 rs cd ab.t b adr.u t bcr. 

10 12 18 17 10 4 3 10 rs cd ab.t b adr.u (cr) ab. 
14 36 45 4 14 17 2 14 rs ad bet r abs.u (ar) bet. 

14 22 10 5 16 20 7 14 rs cd ab.t s ac.u b acst. 
365 162030 rscd ab.t c ar.u r ad. 

11 19 30 4 11 20 10 11 rs cd ab.t (cs) ab.u a bcrt. 

20 34 20 16 20 3 6 20 rs cd ab.t b acs.u b acst. 
16 42 51 4 16 19 4 16 rs ad bet a bdr.u (ar) bet. 

15 25 15 12 15 3 4 15 rs cd ab.t b acs.u b acst. 

8 21 2 25 8 2 9 8 1 rs bd act b adr.u (br) act. 

47 79 37 41 50 15 9 47 rs cd ab.t a bcs.u a best. 
25 66 81 6 25 30 7 25 rs ad bet r abs.u (ar) bet. 
20 30 14 15 24 8 7 20 rs cd ab.t b adr.u b adrt. 



IX. Extension Rules and Infinite Families of 
Inequalities 

The first examples of infinite families of non-Shannon 
inequalities were found by Matus (8). For example, Matus 
showed ((8) Corollary 3, formula (4)) that for any nonnegative 
integer s, the following is an information inequality: 

sI{A-B) <= (s(s + 3)/2)I(A;B\C) (80) 
+ (s(s + l)/2)I(A;C\B)+I(B;C\A) 
+ sI(A;B\D)+sI(C;D). 

Here we have interpreted Matus' variables £1,62, £3, £4 as 
D, C, B, A respectively. Note that in the case s = this is just 
the Shannon inequality < I(B; C\A). If s = 1 we get the 
Zhang-Yeung inequality. When s = 2 this is Inequality ( |37T ). 
But when s = 3 the resulting inequality does not appear 
previously in this paper. Each inequality in this list can be 
proved using the Copy Lemma method, but the number of 
copies goes to 00 as s 00. Matus used this list in a clever 
way to show that T 4 is not polyhedral. 

In this section we will introduce another method of quickly 
generating inequalities. Rather than listing them using an 
index s, we will use a more general approach which we call 
Extension Rules. An Extension Rule turns one inequality of 
a certain type into another of the same type. As an example, 
consider our first Extension Rule. 

Rule [1]: If 

aI(A;B) < bI{A; B\C) + d{A;C\B) + dI(B;C\A) 
+ aI(A;B\D) + fI(A;D\B)+gI(B;D\A) 
+ hI(C;D)+U(C;D\A)+jI(C;D\B) 

is an information inequality with nonnegative coefficients, then 
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so is 

(a + d)I(A; B) < {b + 2d + h + i)I(A; B\C) 

+ (a + c + d + f + g)I(A;C\B) 

+ dI(B;C\A) + (a + d)I(A;B\D) 

+ fI(A-D\B)+gI(B-D\A) 

+ (d + h)I(C;D)+U(C;D\A) 

+ jI(C;D\B). 

Notice that the premise inequality differs from the general 
form given in Section IVIIII in two ways. First, an additional 
term jI(C; D\B) has been added. This of course, is not 
necessary since the coefficient j could be set to zero. In other 
words, this term is only included because it can be and it is 
not known whether it is ever useful. Secondly, notice that the 
coefficient of I(A; B\D) on the right matches the coefficient of 
I(A; B) on the left. This is the only restriction (assuming it is 
of the general form) needed to apply this rule. Notice also that 
the conclusion inequality also satisfies this same restriction, 
allowing us to iterate this rule indefinitely. 

To see why this rule generalizes the Matus list (|80V start 
with the Shannon inequality < I(B; C\A) and iterate the 
rule indefinitely. 

Notice also that there are many other inequalities in Sec- 
tion IVIIII that satisfy the special condition of the coefficients 
and can therefore be iterated using this rule. 

We will now give an outline of the proof of Rule [1] and 
then give several other similar rules. We observe that None of 
the rules discovered so far generalize the second infinite list 
of inequalities given by Matus: 

sI(A;B) <= ( S (s + l)/2)I(A;B\C) (81) 
+ (s(s - 1)/2)J(A; C\B) + 2sI(A; B\D) 
+ sI{A;D\B)+I(B;D\A)+sI(C-D). 

Proof Outline for Ride 1: The procedure for proving 
Rule [1] is similar to that in preceding sections. Given a 
collection of already-known inequalities, one can perform a 
polytope computation to produce a set of extreme rays. Each 
of these extreme rays can be tested against one or more copy 
specifications; if such a test results in a contradiction, then a 
new inequality can be deduced. The difference here is that, 
when testing an extreme ray against a copy specification, one 
can use instances of previously-obtained inequalities, involv- 
ing both the given random variables and the copy variables, 
in an attempt to reach a contradiction. 

For example, suppose we take the Zhang-Yeung inequality 
as known, and produce the list of extreme rays (see (l36l). 
We now consider the copy specification "R is a D-copy of A 
over BC". Using the equations represented by this copy spec- 
ification, the Shannon inequalities on variables A, B, C, D, R, 
and the instances of the Zhang-Yeung inequality obtained by 
substituting in combinations of these five variables, one finds 
that some of the listed extreme rays are not attainable. This 
leads to additional inequalities, such as: 



This inequality could be obtained purely using copy vari- 
ables (and Shannon inequalities); in fact, it is one of the 
six inequalities we obtained in Section [V] using two copy 
variables. This is not surprising, because further investigation 
shows that the process of obtaining this inequality here used 
only one instance of Zhang-Yeung, namely: 

I(AR; BR) < 2I{AR;BR\CR) (83) 
+ I(AR; CR\BR) + I(BR; CR\AR) 
+ I(AR; BR\D) + I(CR; D). 

It only takes one additional copy variable to prove (|83| > 
directly, so two copy variables in all suffice to prove (|82"V 

But further information can be extracted from the proof of 
(l82l which will allow us to get a substantially more general 
result. First, we find that the proof of d8Zt from (183) actually 
shows that the inequality 

21 (A; B) < 5I(A;B\C) + 3I(A;C\B) +I(B;C\A) (84) 
+ 2I(A; B\D) + 2I(C] D) + I(AR; BR) 

- 2I{AR\BR\CR)-I{AR-CR\BR) 

- I(BR; CR\AR) — I(AR; BR\D) — I(CR; D) 

is provable from "i? is a Z?-copy of A over BC" and the 
Shannon inequalities. It turns out that (f84-b can be decomposed 
into several simpler inequalities: each of the inequalities 



I(A-B) 


< 


I(A;C\B) + 


I(A;B\D)+I(AR; BR) 






I{AR; BR\D) 





< 


I(A;B\C)~ 


I(AR; BR\CR) 





< 


I(A;C\B)- 


I(AR; CR\BR) 


I(A;B) 


< 


2I(A;B\C) - 


\- 1 {A; C\B) + I(B; C\A) 




+ 


I(A;B\D) + 


I{C;D) - I{BR;CR\AR) 





< 


I(A;B\C) + 


I(C;D) - I(CR; D) 



is provable from "R is a Z?-copy of A over BC" and the 
Shannon inequalities. (This decomposition was found by a 
trial-and-error approach.) 

Now that we have divided d84T > into these pieces, we can 
reassemble the pieces with different coefficients to get a more 
general rule for deducing inequalities from old ones. By 
putting arbitrary nonnegative coefficients a, b, c, d, h on the 
five inequalities above, instead of the coefficients 1,2,1,1,1 
used to get d84l i. we get: the inequality 

(a + d)I(A;B) < (b + 2d + h)I(A; B\C) 

+ {a + c + d)I{A;C\B)+dI(B;C\A) 

+ (a + d)I(A;B\D) + (d + h)I(C;D) 

+ aI(AR; BR) — bI(AR; BR\CR) 

- d(AR; CR\BR) - dI(BR; CR\AR) 

- aI(AR; BR\D) — hI(CR; D) 

is provable from "i? is a D-copy of A over BC" and the 
Shannon inequalities. Therefore, if 



2I{A-B) < 5I(A;B\C) +3I{A;C\B) (82) 
+I(B; C\A) + 2I(A; B\D) + 2I(C; D). 



aI(A;B) < I(A; B\C) + d(A;C\B) + dI{B;C\A) 
+ aI(A;B\D) + hI(C;D) 



22 



is an information inequality (i.e., always true, so there is no 
problem substituting in AR, BR, CR, D for A, B, C, D), then 

(a + d)I(A;B) < (b + 2d + h)I(A; B\C) 

+ (a + c + d)I(A;C\B)+dI(B;C\A) 
+ (a + d)I(A;B\D) + (d + h)I(C;D) 

is also an information inequality. 

It turns out that we can increase its applicability by adding 
a few more pieces to the list needed for (|84| |. Namely, we use 
the fact that the inequalities 

< I(A;C\B)+I(A;D\B)-I(AR;D\BR) 

< I(A;0\B)+I(B;D\A)-I(BR;D\AR) 

< I(A;B\C)+I(C;D\A)-I(CR;D\AR) 

< I(C;D\B) - I(CR;D\BR) 

are provable from "i? is a D-copy of A over BC" and the 
Shannon inequalities. Using these additional pieces, we can 
extend the above rule to the form Rule [1]. 

Now the premise inequality (assuming it is of the general 
form we have seen earlier) needs to meet only one restriction: 
the coefficient of I(A;B — D) on the right must match the 
coefficient of I(A;B) on the left. (We tried to find another 
piece which would remove this restriction, but did not find a 
suitable one.) ■ 

We now list the other extension rules that have been found, 
where much of the above process has been summarized 
succinctly. Each of the lower case letters in these rules is 
assumed to be a nonnegative real number. 

Rule [2] Given: 

aI(A;B) 

< (b + h)I(A; B\C) + d(A; C\B) + dI(B; C\A) 
+ aI(A;B\D)+fI(A;D\B)+gI(B;D\A) 

+ hI(C;D)+U(C;D\A)+jI(C;D\B) 

Get: 

(a + b)I(A; B) 

< (2b + h)I(A; B\C) + (b + c + h + j)I(A; C\B) 
+ (b + d + h + i)I(B; C\A) + (a + b)I(A; B\D) 
+ fI(A; D\B) + gI(B; D\A) + (b + h)I(C; D) 
+ U(C;D\A)+jI(C;D\B) 

Using: R is copy of C over AB 
Substitution: AR BR CR D 
Rule [3] Given: 

aI(A;B) 
< aI(A;B\C)+d(A;C\B) 

+ (d + h + z)I(B; C\A) + (a + e + h + z)I(A; B\D) 
+ fI(A; D\B) + gI(B; D\A) + {a + h)I(C; D) 
+ U(C;D\A)+jI(C;D\B) 



Get: 

(a + z)I{A; B) 

< (2a + d + h + i + 2z)I(A;B\C) 

+ (a + c + e + f + g + 2h + z)I(A;C\B) 

+ (d + h + z)I(B; C\A) + (a + e + h + z)I(A; B\D) 

+ fI(A; D\B) + gI(B; D\A) + (a + h + z)I(C; D) 

+ U(C;D\A)+jI(C;D\B) 

Using: R is copy of A over BC 
Substitution: AR BR C DR 
Rule [4] Given: 

aI(A; B) 

< aI(A;B\C)+d(A;C\B)+dI(B;C\A) 

+ (e + h)I(A; B\D) + fI(A; D\B) + gI(B; D\A) 
+ hI(C;D)+U(C;D\A)+jI(C;D\B) 

Get: 

(a + e)I(A;B) 

< (a + 2e)I(A; B\C) + (c + e + h + j)I(A; C\B) 
+ (d + e + h + i)I(B; C\A) + (e + h)I(A; B\D) 
+ fI(A; D\B) + gI(B; D\A) + (e + h)I(C; D) 
+ U(C;D\A)+jI(C;D\B) 

Using: R is copy of C over AB 
Substitution: AR BR C DR 
Rule [5] Given: 

aI(A;B) 

< aI(A;B\C)+d(A;C\B) 
+ (d + h + z)I(B;C\A) 

+ (a + h + z + z')I(A;B\D)+fI(A;D\B) 
+ gI{B; D\A) + (a + h)I(C; D) + U(C; D\A) 
+ jI(C;D\B) 

and 

a'I(A; B) 

< b'I(A; B\C) + c'I(A; C\B) + z'I(B; C\A) 
+ a'I(A; B\D) + f'I(A; D\B) + g'I(B; D\A) 
+ h'I(C; D) + i'I(C; D\A) + j'I(C; D\B) 

Get: 

(a + a + z + z')I(A; B) 
< (2a + d + h + i + b' + ti + i' + 2z + 2z) 
1(A-B\C) 

+ (a + c + f + g + 2h + a' + c' + f + g' + z + z') 

I(A; C\B) + (d + h + z + z')I(B; C\A) 
+ (a + h + a + z + z')I(A;B\D) 
+ (f + f')I(A- D\B) + (g + g')I(B; D\A) 
+ (a + h + ti + z + z')I(C; D) 
+ (i + i')I(C- D\A) + (j + j')I(C; D\B) 
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Using: R is copy of A over BC 
Substitutions: AR BR C DR; AR BR CR D 
Rule [6] Given: 

aI(A;B) 

< aI{A;B\C)+d(A;C\B)+dI(B;C\A) 

+ (h + z)I(A; B\D) + fI(A; D\B) + gI(B; D\A) 

+ hI{C;D)+iI(C;D\A)+jI(C;D\B) 

and 

a' I (A; B) 

< (ti + z)I{A;B\C)+c'I(A;C\B) 

+ d'I(B: C\A) + a'I(A; B\D) + f'I(A; D\B) 
+ g'I(B: D\A) + h'I(C; D) + i'I(C; D\A) 
+ j'l(C;D\B) 

Get: 

(a + a + z)I(A; B) 

< {a + ti + 2z)I(A;B\C) 

+ (c + h + j + c' + ti + j' + z)I(A; C\B) 
+ (d + h + i + d' + ti + i + z)I{B\ C\A) 
+ (h + a' + z)I(A; B\D) + (/ + f')I(A; D\B) 
+ (g + g')I(B; D\A) + (h + ti + z)I{C; D) 
+ (i + i')I(C; D\A) + (j + j')I(C; D\B) 

Using: R is copy of C over AB 
Substitutions: AR BR C DR; AR BR CR D 
Rule [7] Given: 

aI(A; B) 

< bI(A;B\C)+d(A;C\B) 

+ (d + j + z)I{B;C\A)+eI(A;B\D) 

+ fI(A; D\B) + (g + x + z)I(B; D\A) + hI(C; D) 

+ U(C;D\A) + (j + x + z)I(C;D\B) 

Get: 

(a + z)I(A; B) 

< (a + b + c + f + j+x + 2z)I(A;B\C) 
+ (-a + b + c+e + j +x + z)I{A;C\B) 

+ {d + j + z)I(B; C\A) + (e + z)I(A; B\D) 

+ fI(A:D\B) + (g + x)I(B;D\A) 

+ (h + j + z)I(C; D) + H(C; D\A) + xI(C; D\B) 

Using: R is copy of G over AB 
Substitution: ARC D 
Rule [8] Given: 

aI(A;B) 

< (h + a' + c' + z)I(A;B\C)+d(A;C\B) 
+ dI(B;C\A) +aI(A;B\D) + fI(A;D\B) 
+ gI(B;D\A) + hI(C;D) + U(C;D\A) 
+ jI(C;D\B) 



and 

a'I{A- B) 

< {a 1 + b')I(A-B\C)+c'I{A-C\B) 

+ (d + h + d'+ f + z)I{B- C\A) + e'I{A] B\D) 

+ f'l(A;D\B) + (a + g + z)I{B;D\A) 

+ tiI(C;D)+i'I(C;D\A) 

+ (j' + z)I(C;D\B) 

Get: 

(a + a' + z)I(A; B) 

< (h + 2a' + b' + c' + f + f + 2z)I{A; B\C) 
+ (c + h + j + b' + c' + e' + f + z)I(A- C\B) 
+ (d + h + i + d' +j' + z)I(B-C\A) 

+ (a + e' + z)I{A- B\D) + (/ + f)I(A; D\B) 

+ (g)I(B;D\A) + (h + ti +f + z)I(C;D) 

+ (i + i')I(C;D\A) + (j)I(C;D\B) 

Using: R is copy of C over AB 
Substitutions: AR BR CR D; A R C D 
Rule [9] Given: 

aI(A;B) 

< (a + b + f + g + z)I(A;B\C)+d(A;C\B) 
+ dI{B;C\A)+eI(A;B\D) + fI(A;D\B) 

+ (g + g 2 + z)I{B:D\A) + hI(C;D) 
+ U(C;D\A) + (j + z)I(C;D\B) 

Get: 

(a + z)I(A; B) 
< (a + b + f + g + 2z)I{A;B\C) 
+ (c + d + e + f + g + g 2 + z)I(A;C\B) 
+ (a + c + d+f + g+j + z)I(B;C\A) 
+ (e + z)I(A;B\D) + (f)I(A;D\B) 
+ (g 2 )I(B;D\A) + (h + z)I(C;D) 
+ (i)I(C;D\A) + (j)I(C;D\B) 

Using: R is copy of A over BC 
Substitution: A R C D 

Rule [10] Given: 

aI(A;B) 

< bI(A;B\C) + (c + x + z)I(A;C\B) 

+ dI(B;C\A)+eI(A;B\D) + fI(A;D\B) 

+ gI(B; D\A) + hI(C; D) + (i + x+ z)I(C; D\A) 

+ (j + z)I(C;D\B) 
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Get: 

(a + z)I(A;B) 

< (b + d + h + i + x + 2z)I(A;B\C) 
+ {c + n+x + z)I{A-C\B) 

+ (b + d + x + z)I(B; C\A) + (e + z)I(A; B\D) 

+ (f)I(A;D\B) + (g)I(B;D\A) 

+ (h + z)I(C;D) + (i)I(C;D\A) 

+ (j 2 )I(C;D\B) 

Using: R is copy of A over BC 
Substitution: A B R D 
Rule [11] Given: 

aI(A;B) 

< bI(A; B\C) + d(A; C\B) + (d + x)I(B; C\A) 
+ (a + d + e)I(A;B\D) + fI(A;D\B) 

+ gI(B;D\A) + aI(C;D)+U(C;D\A) 
+ jI(C;D\B) 

Get: 

(a + d)I(A; B) 

< (a + b + 2d + i + x)I(A;B\C) 

+ (a + c + d + e + f + g)I(A;C\B) 
+ {d + x)I(B;C\A) + (a + d + e)I(A;B\D) 
+ fI(A; D\B) + gI(B; D\A) + (a + d)I(C; D) 
+ U(C;D\A)+jI(C;D\B) 

Using: R is copy of A over BC 
Substitution: AR BR CR DR 
Rule [12] Given: 

aI(A; B) 

< {a + z)I(A; B\C) + d{A; C\B) + dI(B; C\A) 
+ (a + x + z)I{A;B\D) + fI(A;D\B) 

+ gI(B;D\A)+aI{C;D)+U(C;D\A) 
+ jI(C;D\B) 

Get: 

(a + z)I(A;B) 

< (a + 2z)I(A; B\C) + (a + c + j + z)I(A; C\B) 
+ (a + d + i + x + z)I(B;C\A) 

+ (a + x + z)I(A;B\D) + fI(A;D\B) 

+ gI(B; D\A) + (a + z)I(C; D) + H(C; D\A) 

+ jI(C;D\B) 

Using: R is copy of C over AB 
Substitution: AR BR CR DR 
Rule [13] Given: 

aI(A;B) 

< (a + y + z)I(A;B\C)+cI(A;C\B) 

+ dI(B; C\A) + (a + z)I(A; B\D) + fI(A; D\B) 
+ gI(B;D\A)+aI(C;D) + U(C;D\A) 
+ jI(C;D\B) 



a'I(A; B) 

< b'l(A; B\C) + c'I(A; C\B) + d'l(B; C\A) 
+ (h' + y)I(A;B\D) + f'l(A;D\B) 

+ g'I(B; D\A) + h'I(C; D) + i'l(C; D\A) 
+ j'I(C;D\B) 

Get: 

(a + a + y + z)I(A; B) 

< {a + b 1 + i' + 2y + 2z)I(A;B\C) 

+ (a + c + j + c' + f' + h' + i'+j' + y + z) 
I{A-C\B) 

+ {a + d + i + d' +g' + ti + i' + y + z)I(B; C\A) 
+ (a + h' + y + z)I(A; B\D) + (f + f')I(A; D\B) 
+ (g + g')I(B; D\A) + (a + h' + y + z)I{C; D) 
+ (i + i')I(C; D\A) + (j + j')I(C; D\B) 

Using: R is copy of C over AB 
Substitutions: AR BR CR DR; ABC DR 
Rule [14] Given: 

aI(A;B) 

< (a + b' + c' + z)I{A;B\C) +d(A;C\B) 
+ dI(B;C\A) + (a + e' + f + z)I(A;B\D) 
+ fI(A;D\B)+gI(B;D\A)+aI(C;D) 

+ iI(C;D\A)+jI(C;D\B) 

and 

a'I(A;B) 
< b'I(A;B\C)+c'I(A;C\B) 
+ (a + d + i+f + z)I(B; C\A) + e'I(A; B\D) 
+ f'I(A;D\B) + (a + g + i + z)I{B;D\A) 
+ h'I(C;D)+i'I(C;D\A) 
+ (j' + z)I(C;D\B) 

Get: 

(a + a' + z)I(A; B) 

< (a + a+b' + c' + f + j' + 2z)I{A; B\C) 

+ (a + c + j -a' + b' + c' + e' + j' + z)I(A; C\B) 

+ (a + d + i+f + z)I{B;C\A) 

+ {a + e' + z)I(A; B\D) + (f + f')I(A; D\B) 

+ gI{B;D\A) + (a + ti +f + z)I(C;D) 

+ (i + i')I(C;D\A)+jI(C;D\B) 

Using: R is copy of C over AB 
Substitutions: AR BR CR DR; A R C D 
Rule [15] Given: 

aI(A; B) 

< aI(A;B\C)+d(A;C\B)+dI(B;C\A) 
+ (h + a')I(A;B\D) + fI(A;D\B) 
+ gI(B;D\A) + hI(C;D)+U(C;D\A) 
+ jI(C;D\B) 
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a' I (A; B) 
< b'I(A;B\C)+dI(A;C\B) 
+ (a + d')I(B; C\A) + (a' + e')I(A; B\D) 
+ f'l(A;D\B) + (g + h + g')I(B;D\A) 
+ h'I(C; D) + i'I(C; D\A) + j'I(C; D\B) 

Get: 

(a + a')I(A; B) 

< (a + a' + b' + d + f' + j')I(A;B\C) 

+ (c + h + j + b' + d + d + j')I(A; C\B) 
+ (d + h + i + d 1 + g[ + j')I(B;C\A) 
+ (h + a 1 + e')I{A- B\D) + (f + f)I(A; D\B) 
+ (g + g' 2 )I(B- D\A) + (h + ti + f)I(C; D) 
+ (i + i')I(C;D\A)+jI(C;D\B) 

Using: R is copy of C over AB 
Substitutions: AR BR C DR; A R C D 
Rule [16] Given: 

aI(A; B) 

< aI(A;B\C)+d(A;C\B) 

+ (-a + a' + d + d' + /' + x)I(B;C\A) 
+ (e' + f' + x)I(A;B\D)+fI(A;D\B) 
+ gI(B; D\A) + (x - w)I(C; D) + U(C; D\A) 
+ jI(C;D\B) 

and 

a'I(A; B) 

< (a' + b' + f + x)I{A; B\C) + c'I(A; C\B) 
+ d'I(B; C\A) + e'I(A; B\D) + f'I(A; D\B) 

+ (g + x)I(B; D\A) + tiI(C; D) + i'I(C; D\A) 
+ (j' + w)I(C;D\B) 

Get: 

{a + a + w)I(A; B) 

< {a + i + a +b' + /' + w + x)I(A;B\C) 

+ (-a + c + f + g + c' + d' + e' + f - w + 2x) 
I(A;C\B) 

+ (-a + a' + c' + d' + f + f + x)I{B- C\A) 
+ (e ; + x)I(A; B\D) + (/ + f')I(A; D\B) 
+ (g)I(B;D\A) + (h' + x)I(C;D) 
+ (i + i')I(C; D\A) + (j + f)I(C; D\B) 

Using: R is copy of A over BC 
Substitutions: AR BR C DR; A R C D 
Rule [17] Given: 

aI(A; B) 

< (h + z)I(A; B\C) + d(A; C\B) + dI(B; C\A) 
+ eI{A;B\D) + fI(A;D\B)+gI{B;D\A) 

+ hI(C;D)+H(C;D\A)+jI(C;D\B) 



a'I(A; B) 

< b'l{A; B\C) + c'l(A; C\B) + d'l(B; C\A) 
+ (ti + z)I(A;B\D) + f'I(A;D\B) 
+ g'I(B;D\A) + h'I(C;D)+i'l(C;D\A) 
+ j'I(C;D\B) 

Get: 

(a + a' + z)I(A; B) 
< (h + b' + i' + 2z)I{A;B\G) 

+ (2c + h + j +c' + f' + h' + i' + ]' + z)I(A; C\B) 

+ (2d + h + i + d' + g' + ti +i' + z)I(B; C\A) 

+ (e + ti + z)I(A; B\D) + (/ + f)I(A; D\B) 

+ (g + g')I(B; D\A) + (h + ti + z)I(C; D) 

+ (i + i')I(C; D\A) + (j + j')I(C; D\B) 

Using: R is copy of C over AB 
Substitutions: A B CRD; A B C DR 
Rule [18] Given: 

aI(A; B) 

< bI(A;B\C) + (i + b' + d + z)I(A;C\B) 
+ dI(B;C\A)+eI(A;B\D) 
+ (d + z)I(A- D\B) + gI(B; D\A) + hI(C; D) 
+ (i + Zl )I(C;D\A)+jI(C;D\B) 

and 

a'l(A;B) 
< b'I(A;B\C)+dI(A;C\B) 
+ (b + j' + z)I(B;C\A) + dl(A;B\D) 
+ f'I(A;D\B) + (e + g + z)I(B;D\A) 
+ h'I(C;D)+i'I(C;D\A) 
+ (j' + z 2 )I(C;D\B) 

Get: 

(a + a +z)I(A;B) 

< (a + b + d + g + i + a + b' + d + /' + j' + 2z) 
I(A;B\C) 

+ (i-a' + b' + d + d + f + z)I(A- C\B) 
+ (-a + b + d + e + i+j' + z)I(B;C\A) 
+ (e + d + z)I(A-B\D) + f'I(A-D\B) 
+ (g)I(B; D\A) + (h + i + ti + j' + z)I(C; D) 
+ i'I(C;D\A)+jI{C;D\B) 

Using: R is copy of C over AB 
Substitutions: R B C D; A R C D 
Rule [19] Given: 

aI(A- 1 B) 

< (a + Zl )I{A;B\C) +d(A;C\B) 

+ dI(B; C\A) + (a + z)I(A; B\D) + fI(A; D\B) 
+ gI(B;D\A)+aI(C;D) + U(C;D\A) 
+ jI(C;D\B) 



26 



and 



Get: 



a' I (A; B) 

< (ti + z 2 )I(A;B\C)+c'I(A;C\B) 
+ d'I(B; C\A) + e'I(A; B\D) + f'l(A; D\B) 
+ g'I(B; D\A) + h'I(C; D) + i'I(C; D\A) 
+ j'I(C;D\B) 



Get: 



(a - 


ha' + z)I(A]B) 




< (a- 


hti + 2z)I(A;B\C) 




+ (a- 


h c + j + 2c + ti + j 


1 + z)I(A-C\B) 


+ (a- 


hd + i + 2d' + ti + i 


+ z)I{B-C\A) 


+ (a- 


Ve' + z)I{A;B\D) + 


(f + f')I(A;D\B) 


+ {g- 


hg')^(-B;Z?|A) + (a^ 


- ti + z)I(C; D) 


+ (i 4 


-i')I(C;D\A) + (j + 


j')I(C;D\B) 



Using: R is copy of C over AB 
Substitutions: AR BR CR DR; A B CR D 
Rule [20] Given: 

aI(A; B) 

< (a + b)I(A;B\C)+d(A;C\B) 
+ (j + x + z)I(B;C\A)+eI(A;B\D) 
+ fI(A;D\B)+xI(B;D\A) + hI(C;D) 
+ U(C;D\A) + (j + x)I(C;D\B) 

Get: 

(a + x + z)I(A; B) 
< (a + b + x + 2z)I(A; B\C) + (c + z)I(A; C\B) 
+ (j + z)I(B;C\A) 

+ (a + c + e + / +j + 2x + z)I{A;B\D) 

+ (b + e + f + j + x)I(A; D\B) + (x)I(B; D\A) 

+ (h + x + z)I{C;D) + (i)I(C;D\A) 

+ (j)I(C;D\B) 

Using: RS is copy of CD over AB 
Substitution: A D R S 
Rule [21] Given: 

aI(A; B) 

< (a + b)I(A;B\C)+d(A;C\B) 

+ (j + w + x)I(B;C\A) +eI(A;B\D) 

+ fI(A; D\B) + (g + x)I(B; D\A) + hI(C; D) 

+ U(C;D\A) + (j + x)I(C;D\B) 



and 



a'I(A; B) 
< (a' + b')I(A;B\C) +c'l(A;C\B) 
+ zI(B; C\A) + e'I{A; B\D) + f'l(A; D\B) 
+ (g' +f -w + z)I(B;D\A) + til(C;D) 
+ i'I(C;D\A) + (j'+z)I(C;D\B) 



(a + a' + x + z)I(A; B) 

< (a + b + j + 2a' + b' + c' + f' + x + 2z)I(A; B\C) 

+ (c + b' + c' + e' +f + z)I{A;C\B) 

+ (f + z)I(B;C\A) 

+ (a + c + e + f + e'+j' + 2x + z)I(A;B\D) 

+ (b + e + f + f'+j + x)I(A;D\B) 

+ (g + g' + j + x)I(B;D\A) 

+ (h + j + ti +j' + x + z)I(C;D) 

+ (i + i')I(C;D\A) 

Using: RS is copy of CD over AB 
Substitutions: A D R S; A C R S 
Rule [22] Given: 

al(A-B) 

< (h+ z)I(A; B\C) + d(A; C\B) + dI(B; C\A) 
+ aI(A;B\D) + fI(A;D\B)+gI(B;D\A) 
+ hI(C;D)+U(C;D\A)+jI(C;D\B) 

and 

a'l(A;B) 
< {a' + b')I(A;B\C) +c'I(A;C\B) 
+ (a + f + z)I(B;C\A) + e'I(A;B\D) 
+ fI(A; D\B) + g'I{B- D\A) + tiI(C; D) 
+ i'l(C;D\A)+fl(C;D\B) 



Get: 



(a + a' + z)I(A; B) 

< {h + a' + b' +j' + 2z)I{A;B\C) 
+ (c + h + j + c' + z)I(A;C\B) 

+ (d + h + i + z)I(B;C\A) 

+ (a + a' + c' + e' + f' + z)I(A;B\D) 

+ (f + b' + e' + f'+j')I(A;D\B) 

+ (g + g'+j')I(B;D\A) 

+ (h + h'+ f + z)I(C; D) + (i + i')I(C; D\A) 

+ jI(C;D\B) 

Using: RS is copy of CD over AB 
Substitutions: AC BC CR S; A D R S 
Rule [23] Given: 

aI(A; B) 

< (a + b + f + g + x)I(A; B\C) + d(A; C\B) 
+ dI(B;C\A)+eI(A;B\D) + fI{A;D\B) 

+ (g + 92)I{B\ D\A) + hI(C; D) + U(C; D\A) 
+ (j + x)I(C;D\B) 
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Get: 

(a + 2x)I{A] B) 

< (a + b + f + g + 3x)I{A;B\C) 

+ (c + d + e + f + g + g 2 +ix)I(A;C\B) 

+ (a + c + d + f + g + j + 2x)I(B;C\A) 

+ (e + 3x)I(A;B\D) + (f + x)I(A;D\B) 

+ (g 2 )I(B;D\A) + (h + 2x)I(C;D) 

+ (i)I(C;D\A)+jI(C;D\B) 

Using: R is copy of A over BC, S is copy of A over BDR 
Substitution: ARC D 
Rule [24] Given: 

(w - x)I(A; B) 

< wI(A;B\C)+d(A;C\B)+dI{B;C\A) 

+ {a' + b' + c' + w + z)I{A; B\D) + fI(A; D\B) 

+ gI(B; D\A) + (w - x)I(C\ D) + U(C; D\A) 

+ jI(C;D\B) 

and 

a'l(A;B) 

< {a' + b')I(A;B\C) + c'I(A;C\B) 

+ (g + i+f + w + z)I(B; C\A) + e'I(A; B\D) 

+ f'I(A;D\B) + (g' + z)I(B;D\A) 

+ til(C;D) +i'l(C;D\A) 

+ 0" + z)I(C;D\B) 

Get: 

(a' + w + z)I{A; B) 

< (a' + b' +f + w + x + z)I{A-B\C) 
+ (c + j + c' + w)I(A;C\B) 

+ {d + i + w)I(B;C\A) 

+ {a' + c' + e' + f + w + 2z)I(A;B\D) 

+ (f + b' + e' + f'+j' + z)I(A;D\B) 

+ (g + g' + j' + z)I(B;D\A) 

+ (ti + f + w + z)I(C; D) + {i + i')I(C; D\A) 

+ jI(C;D\B) 

Using: RS is copy of CD over AB 
Substitutions: AC BC CR CS; A D R S 
Rule [25] Given: 

aI(A;B) 

< bI(A;B\C)+d(A;C\B)+dI(B;C\A) 

+ (h + z)I(A; B\D) + fI(A; D\B) + gI(B; D\A) 
+ hI{C;D) + U(C;D\A)+jI(C;D\B) 

Get: 

{a + z)I(A;B) 

< (b + z)I(A; B\C) + {c+f + h + j)I(A; C\B) 
+ {d + g + h + i)I{B- C\A) + (h + 2z)I{A; B\D) 
+ (f + z)I(A;D\B) + (g + z)I(B;D\A) 

+ (h + z)I(C; D) + (i)J(C; D\A) + jI(C; D\B) 



Using: RS is copy of CD over AB 
Substitution: ABC DR 
Rule [26] Given: 

aI(A; B) 
< bI(A;B\C)+d(A;C\B) 
+ (j + e' + f' + w + x + z)I(B;C\A) 
+ (a + e)I{A;B\D) + fI(A;D\B) 
+ (g + w)I(B; D\A) + hI(C; D) + U(C; D\A) 
+ (j + w + x)I(C;D\B) 

and 

a'I(A; B) 
< (a' + b')I(A;B\C) +c'I(A;C\B) 
+ zI(B; C\A) + e'l(A- B\D) + fl(A; D\B) 
+ (b + c + g' +f + w + x + z)I(B;D\A) 
+ h'I(C;D)+i'I(C;D\A) 
+ (j'+z)I(C;D\B) 

Get: 

(a + a' + w + z)I(A; B) 
< (b + j + 2a' + b' + c' + f + w + 2z)I{A- B\C) 
+ (c + f + z)I(A;C\B) 
+ (d + b' + c' + e'+j'+x + z)I(B;C\A) 
+ (2a + c + e + / + e' + f + 2w + x + z)I(A; B\D) 
+ (b + e + f +j +g' + w + x)I{A;D\B) 
+ (g+j + f + w)I(B;D\A) 
+ (h + j + h'+j' + w + z)I(C; D) + i'I(C; D\A) 
+ xI(C;D\B) 

Using: RS is copy of CD over AB 
Substitutions: A D R S; B C R S 
Rule [27] Given: 

aI{A-B) 

< bI(A; B\C) + d(A; C\B) + (d + z)I{B; C\A) 
+ eI(A; B\D) + fI(A; D\B) + (x + z)I(B; D\A) 
+ hI(C; D) + U(C; D\A) + (x + z)I(C; D\B) 

Get: 

(a + 2x + z)I{A; B) 

< (a + b + c + f + 5x + 2z)I(A;B\C) 
+ (-a + b + c + e + 3x + z)I(A;C\B) 

+ (d + x + z)I{B; C\A) + (e + 2x + z)I(A; B\D) 
+ (f)I{A;D\B) + (h + 2x + z)I(C;D) 
+ (i)I{C-D\A) 

Using: R is copy of C over AB S is copy of R over AC 
Substitution: A RC D 
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Rule [28] Given: 



Get: 



and 



Get: 



aI(A\B) 
< bI{A;B\C)+d(A;C\B) 
+ (j + g' + h' + i' + z)I(B;C\A) 
+ (a + e)I(A; B\D) + fI{A; D\B) + zI(B; D\A) 
+ hI(C; D) + U(C; D\A) + (j + z)I(C; D\B) 



a'l(A;B) 

< b'I(A; B\C) + c'I(A; C\B) + d'I(B; C\A) 
+ (b + c + j + ti + z)I{A; B\D) + f'I(A; D\B) 
+ g'I(B; D\A) + h'I(C; D) + i'I{C; D\A) 
+ j'I(C;D\B) 



(a + a +z)I{A;B) 

< (b + b' + z)I{A-B\C) 

+ (c' + f + h'+j')I(A;C\B) 

+ (j + d'+ g' + h' + i')I(B; C\A) 

+ (2a + e + f + j + ti + 2z)I(A;B\D) 

+ (b + e + f + j + f + z)I(A;D\B) 

+ (g' + z)I(B;D\A) + (h + ti + z)I(C;D) 

+ (i + i')I(C; D\A) + + j')I(C; D\B) 



Using: RS is copy of CD over AB 
Substitutions: A D R S; A B C DR 
Rule [29] Given: 

aI(A;B) 

< bI(A;B\C)+xI(A;C\B)+dI(B;C\A) 

+ eI(A; B\D) + (/ + x)I(A; D\B) + gI{B; D\A) 
+ hI(C;D)+xI(C;D\A)+jI(C;D\B) 

Get: 

(a + 2x)I(A; B) 
< (2b + d+e + g+ 5x)I(A; B\C) + (x)I(A; C\B) 
+ (b + 2d + e + g + 3x)I(B;C\A) 
+ (e + 2x)I(A;B\D) + fI(A;D\B) 
+ gI(B; D\A) + (h + 2x)I(C; D) + jI(C; D\B) 

Using: R is copy of A over BC S is copy of R over AB 
Substitution: S B C D 
Rule [30] Given: 

aI(A;B) 

< (a + g + x)I(A;B\C)+d(A;C\B) 

+ dI(B; C\A) + eI(A; B\D) + (f + x)I(A; D\B) 
+ gI(B;D\A) + hI(C;D)+xI(C;D\A) 
+ jI(C;D\B) 



(a + 2x)I(A; B) 

< (a + g + 3x)I(A;B\C) 

+ {a + 2c + d + e + f + 2g + 3x)I(A;C\B) 
+ (a + c + 2d + e + j + 2g + 3x)I(B; C\A) 
+ (e + f + 2x)I(A;B\D) + (g)I(B;D\A) 
+ (h + 2x)I(C;D)+jI(C;D\B) 

Using: R is copy of A over BC S is copy of R over AC 
Substitution: S B C D 
Rule [31] Given: 

aI(A;B) 

< bI(A;B\C)+d(A;C\B)+xI(B;C\A) 

+ (a + x)I(A; B\D) + fI(A; D\B) + gI(B; D\A) 
+ aI(C;D)+U(C;D\A)+xI(C;D\B) 

Get: 

(a + 2x)I{A- B) 
< (a + b + i + 4x)I(A;B\C) 
+ (2a + b + 2c+2f + 2g + 4x)I(A;C\B) 
+ (c + x)I(B;C\A) + (a + 2x)I(A;B\D) 
+ (b + c+ f + g + x)I(A;D\B) 
+ (a + f + g + x)I{B; D\A) + {a + 2x)I(C; D) 
+ {i)I(C;D\A) 

Using: R is copy of A over BC S is copy of A over BDR 
Substitution: AR RS CR DR 
Rule [32] Given: 

(x - w)I{A; B) 

< xI(A;B\C)+d(A;C\B) + dI{B;C\A) 

+ (x + z)I(A; B\D) + fI(A; D\B) + gI(B; D\A) 

+ (x- w)I(C; D) + U(C; D\A) + jI(C; D\B) 

Get: 

(x + z)I(A; B) 

< (w + x + z)I{A- B\C) + (c + j + x)I(A; C\B) 
+ (d + i + x)I{B;C\A) + (x + 2z)I(A;B\D) 

+ (f + z)I(A;D\B) + (g + z)I(B;D\A) 

+ (x + z)I{C- D) + (i)I(C; D\A) + jI(C; D\B) 

Using: RS is copy of CD over AB 
Substitution: AC BC CR CS 
Rule [33] Given: 

aI(A;B) 

< bI(A;B\C)+d(A;C\B)+dI(B;C\A) 
+ eI(A;B\D) + fI(A;D\B)+xI(B;D\A) 
+ hI(C;D) +xI(C;D\A) +xI(C;D\B) 



29 



Get: 

(a + 2x)I(A; B) 

< (b + e + 4x)I(A;B\C) 

+ (c + f + h + 2x)I(A; C\B) + (d + x)I(B; C\A) 

+ (e + / + h + 3x)I(A;B\D) 

+ (e + / + x)I(A;D\B) + (h + 2x)I(C;D) 

Using: R is copy of C over AB S is copy of R over AD 
Substitution: A B C S 
Rule [34] Given: 

(a-d)I(A;B) 

< (a-w + x)I(A;B\C) + (c + x)I(A;C\B) 

+ dI(B; C\A) + (a + x)I{A; B\D) + fI(A; D\B) 

+ gI(B; D\A) + (a - d)I(C; D) + xI(C; D\A) 

+ jI(C;D\B) 

Get: 

(a + 2x)I(A] B) 

< (a + d + 3x)I(A; B\C) + (a + j + 2x)I(A; C\B) 

+ (a + d + 2x)I(B; C\A) + (a + 2x)I(A; B\D) 

+ (a + f + g + x)I(A;D\B) 

+ (a + d + f + g-w + 2x)I(B;D\A) 

+ (a + 2x)I(C;D)+jI(C;D\B) 

Using: R is copy of C over AB S is copy of B over ADR 
Substitution: RS BR CR DR 
Rule [35] Given: 

(e - w)I(A; B) 

< (e + x)I(A; B\C) + d(A; C\B) + dI(B; C\A) 
+ eI(A; B\D) + fI(A; D\B) + gI(B; D\A) 

+ (e - w)I(C; D) + U(C; D\A) + jI(C; D\B) 

and 

(&' -w')I{A;B) 
< b'I(A; B\C) + c'I(A; C\B) + d'l(B; C\A) 
+ (b'-x)I(A;B\D)+f'I(A;D\B) 
+ g'I(B;D\A) + (b' -w')I(C;D) 
+ i'I(C;D\A)+j'I(C;D\B) 

Get: 

(e + b')I(A;B) 
< (e + b' + w')I(A; B\C) 
+ (c + b 1 +c' +j')I(A;C\B) 
+ (d + b' + d' + i')I(B;C\A) 
+ (e + w + b')I(A;B\D) 
+ (e + f + j + f)I(A;D\B) 
+ (e + g + i + g')I(B; D\A) + (e + b')I(C; D) 
+ (i + i')I(C; D\A) + (j + j')I(C; D\B) 

Using: RS is copy of CD over AB 
Substitutions: AD BD DR DS; AC BC CR CS 



Rule [36] Given: 
aI{A-B) 

< (a + x)I(A; B\C) + d(A; C\B) + dI(B; C\A) 
+ (a + z)I(A; B\D) + fI(A; D\B) + gI(B; D\A) 
+ aI(C;D)+U(C;D\A)+jI(C;D\B) 

and 

a'I(A;B) 

< b'I(A; B\C) + c'I(A; C\B) + d'l(B; C\A) 
+ (ti -x + z)I(A;B\D) + f'I(A;D\B) 

+ g'l(B;D\A) + h'l(C;D)+i'l(C;D\A) 
+ j'I(C;D\B) 

Get: 

(a + a' + z)I(A; B) 

< (a + b' + z)I(A;B\C) 

+ (c + c' + f + h'+j')I(A;C\B) 

+ (d+d' + g' + h' + i')I(B;C\A) 

+ (a + ti + 2z)I(A;B\D) 

+ (a + f + j + f' + z)I(A;D\B) 

+ (a + g + i + g' + z)I(B;D\A) 

+ (a + h' + z)I(C;D) + (i + i r )I(C;D\A) 

+ (j+j')I(C;D\B) 

Using: RS is copy of CD over AB 
Substitutions: AD BD DR DS; A B C DR 

Rule [37] Given: 

aI(A; B) 

< {a + b' + c' +f + z + z')I{A;B\C) 
+ d(A; C\B) + dI(B; C\A) 

+ (a + e' + /' + z')I(A;B\D)+fI(A;D\B) 
+ gI(B;D\A)+aI(C;D)+U{C;D\A) 
+ jI(C;D\B) 

and 

a'I(A;B) 

< b'I(A;B\C)+c'I(A;C\B) 

+ (a + d + i+j' + z + z')I(B;C\A) 
+ e'I(A;B\D) + f'I(A;D\B) 
+ (a + g + i + z')I(B;D\A) + h'l(C;D) 
+ i'l(C;D\A) + (j' + z')I(C;D\B) 
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Get: 



(a - 


ho'- 


f z + z')I{A- B) 


< (a- 


h*>'- 


^j' + 2z + z')I(A;B\C) 


+ (c H 


-c'H 


- z)I(A; C\B) + (d + z)I{B; C\A) 


+ (o- 


ha'- 


fc' + e'+/' + z+ 2z')I(A; B\D) 


+ (a- 


h/H 


-j-a' + b' + e' + f' + f + z')I{A; D\B) 


+ (a- 




-i + j' + z')I(B;D\A) 


+ (o- 


h/i' 


±f+z + z')I(C;D) 


+ (i4 





Using: RS is copy of CD over AB 
Substitutions: AD BD DR DS; A D R S 
Rule [38] Given: 



aI{A-B) 

< (a + x)I(A;B\C) + (x-w)I(A;C\B) 
+ dI(B;C\A)+eI(A;B\D)+xI(A;D\B) 
+ gI(B;D\A)+aI(C;D)+xI(C;D\A) 
+ jI(C;D\B) 

Get: 

(a + 2x)I(A; B) 
< (2a + d + g + 4x)I(A;B\C) 
+ (2j + x)I(A; C\B) + (a + d + e + 3x)I(B; C\A) 
+ (a + e + j + 3x)I(A;B\D) + ()I(A;D\B) 
+ {a + d + g-w + 2x)I(B; D\A) + (a + 2x)I(C; D) 
+ ()I(C;D\A)+jI(C;D\B) 

Using: R is copy of C over AB S is copy of B over ADR 
Substitution: RS BS CS DS 
Rule [39] Given: 

aI(A;B) 

< (o + / + x)I{A;B\C)+d{A;C\B) 

+ dI(B;C\A)+eI(A;B\D) + fI(A;D\B) 
+ xI(B;D\A) + hI(C;D)+U(C;D\A) 
+ xI(C;D\B) 

Get: 

(2a + c + / + 2x)I{A\ B) 
< (3a + 2c + 3/ + 5x)I(A;B\C) 
+ (a + 2c + 2d + 2e + 3f + 3x)I(A;C\B) 
+ (a + c + d + f + x)I(B;C\A) 
+ (a + c + e + f+ 2x)I(A; B\D) + (f)I(A; D\B) 
+ Ql(B;D\A) + (a + c+f + h + 2x)I(C;D) 
+ (i)I(C;D\A)+I(C;D\B) 

Using: R is copy of A over BC S is copy of AR over BC 
Substitution: A S C D 



Rule [40] Given: 

al(A-B) 

< bI(A;B\C)+xI(A;C\B)+dI(B;C\A) 
+ eI(A;B\D) + fI(A;D\B)+gI(B;D\A) 
+ hI(C;D)+xI(C;D\A)+xI(C;D\B) 

Get: 

{a + b + 2x)I{A;B) 

< (3b + 2d + 2h + 5x)I(A;B\C) 

+ (b + 3x)I(A;C\B) + (b + d + x)I(B;C\A) 
+ (b + e + 2x)I(A;B\D) + {f)I(A;D\B) 
+ (g)I(B;D\A) + (b + h + 2x)I(C;D) 

Using: R is copy of A over BC S is copy of AR over BC 
Substitution: A B S D 

Rule [41] Given: 

(a + x)I(A\ B) 
< (a + b + g + x)I(A;B\C) + d(A;C\B) 
+ (d + x)I(B;C\A) + {d + x)I(A;B\D) 
+ fI{A- D\B) + gI(B; D\A) + (a + x)I(C; D) 
+ U(C;D\A)+jI(C;D\B) 

Get: 

(a + b + d + 2g + 2x)I(A; B) 
< (2a + b + 2d + 4g + i + 3x)I(A;B\C) 
+ (2a + 3b + 2c + 2d + f + 4g + i + 5x)I(A; C\B) 
+ {d + g + x)I(B;C\A) 
+ (a + 2b + d + 2g + i + 3x)I(A;B\D) 
+ (a + b + c+d + f + g + 2x)I(A;D\B) 
+ (b + g)I(B; D\A) + {a + b + d + 2g + 2x)I(C; D) 
+ (i)I(C;D\A)+jI(C;D\B) 

Using: R is copy of A over BC S is copy of A over BDR 
Substitution: ARS BRS CRS DRS 

Rule [42] Given: 

aI(A;B) 

< (a + b + g)I{A;B\C)+d(A;C\B) 
+ (d + d')I(B;C\A) 

+ (a + 2d + d' + e)I(A;B\D) + fI(A;D\B) 
+ gI(B;D\A) +aI(C;D) +U(C;D\A) 
+ jI(C;D\B) 
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Get: 

(2a + 6 + 2d + d! + e + g)I(A; B) 
< (3a + 6 + 3d + 1.5a" + 2e + g + i)I(A\ B\C) 
+ (4a + 36 + 2c + 4d + 2.5a" + e + f + 4g + i) 

I(A; C\B) + (a + 2d + a" + e)J(B; C|A) 
+ (3a + 26 + 3d + 1.5d' + e + 2g + i)J(^4; B|Z>) 
+ (a + 6 + c + d + d' + / + <?)/(A;D|B) 
+ (6 + g)I{B; D\A) 
+ (2a + 6 + 2d + d' + e + g)I(C; D) 
+ (i)I(C;D\A)+jI(C;D\B) 

Using: R is copy of A over BC S is copy of A over BDR 
Substitution: ARS BRS CRS DRS 
Rule [43] Given: 

aI(A;B) 

< bI(A;B\C)+d(A;C\B)+zI(B;C\A) 
+ eI{A;B\D) + fI(A;D\B) 
+ (6' + d' + z)I(B;D\A) + hI(C;D) 
+ U{C;D\A) + zI(C;D\B) 

and 

a' I (A; B) 

< b'l(A; B\C) + c'l(A; C\B) + d'I(B; C\A) 
+ e'I(A; B\D) + f'I(A; D\B) + g'I(B; D\A) 
+ h'l(C; D) + i'l(C; D\A) + j'l{C; D\B) 

{a + a! + z)I{A;B) 

< (a + b + c+ f + b' + 2z)I(A;B\C) 
+ (-a + b + c + e + c' + z)I(A;C\B) 

+ {d! + z)I(B; C\A) + (e + e' + z)I(A; B\D) 

+ (f + f')I(A;D\B) 

+ (-a' + 6' + e' +g' + i')I(B: D\A) 

+ (h + ti + z)I(C;D) + (i + i')I(C;D\A) 

+ (j')I(C;D\B) 

Using: RS is copy of CD over AB 
Substitutions: A C R S; AD BRS 

X. Connection with inequalities of Xu, Wang, and 

Sun 

Another, very similar, method that automatically generates 
non-Shannon inequalities was presented by Xu, Wang, and 
Sun ifTOll . As an example of their method, they give four new 
inequalities and also a separate infinite list of inequalities. 

The new inequalities given iflOl . Section V, are summarized 
in the following two theorems. The first theorem has four 
inequalities that were presented as examples of their method 
in Sections V.A and V.B of iflOll . In order to connect these 
with the present work, we have relabeled their variables 
Cii^2,^3,^4 as B, C, A, D in the first three of these inequali- 
ties and as A, B, C, D in the fourth. The inequalities are then 



Get: 



rewritten in the form of Theorem |6] It should be emphasized 
that these inequalities were intended only as examples of their 
method and should not be considered complete or exhaustive. 
In the second theorem below we present their infinite list 
of inequalities from Section V.C of of IflOl . The variables 
£ij £2, £3, £4 are again relabeled as A, B, C, D and the inequal- 
ities are rewritten to match Theorem |6] 

Theorem 8 (Xu, Wang, Sun): The following is an informa- 
tion inequality 

aI(A;B) < bI(A; B\C) + d(A;C\B) + dI(B;C\A) 
+ eI(A;B\D) + fI(A;D\B)+gI(B;D\A) 
+ hI{C;D)+iI(C;D\A) 

for each of the following values of (a, 6, c, d, e, /, g, h, i). 

(8, 12, 33, 10, 8, 15, 1, 8, 0) (85) 

(5, 7, 20, 5, 6, 9, 1, 5, 0) (86) 

(4, 5, 17, 6, 6, 7, 0, 4, 0) (87) 

(3, 4, 6, 6, 3, 1, 1, 3, 0) (88) 

Theorem 9 (Xu, Wang, Sun): For each positive integer s, 
the following is an information inequality 



(2 s 



1)I(A;B) < 2 S - 1 I(A;B\C) 



where 



+ (c 1 -2*- 1 )I(A;C\B) 

+ ( Cl - 2 S - 1 )/(S;C7|A) 

+ (2 s - 1 -1)I(A;B\D) 

+ {c 2 ~2 s ~ 1 + 1)I(A;D\B) 

+ {c 2 -2 s - 1 + \)I{B;D\A) 

+ (2 S - 1 -1)/(C;C), 



(5 + + 5_)/4 

((v^-l)S+-(v^+l)S_)/4 



(2 + V2~y 

(2 - V2) s 



Cl 

S+ 



In this section we will show how to derive the inequalities 
of Xu, Wang, and Sun by adding together linear combinations 
of inequalities from the present work. This will usually result 
in stronger versions of these inequalities. 

To begin, first note that Inequality d87l i is identical to d63l l. 
It does not appear in the list given in Section IVIIII because 
it has been superseded. We also note that Inequality d88l is a 
weaker form of 



(3- 



4, 4, 



1, 1, 



0) 



63, 



which has also been superseded. 

To get Inequality d85l l. we add together the previous in- 
equalities 

(3, 4, 6, 6, 3, 0, 0, 3, 0) 03) 
(4, 6, 4, 3, 4, 2, 1, 4, 0) ((54) 
(2, 4, 1, 2, 2, 3, 0, 2, 0)(l/2) <E3>. 
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This gives us the inequality 

(8, 12, 21/2, 10, 8, 7/2, 1, 8, 0) dH 

which is an improvement of d85l l. 

To get Inequality d86l l. we first switch the variables C and 
D in the inequality 

(3, 7, 5, 1, 3, 1, 1, 3, 0) (ED 

to obtain the inequality 



(3, 3, 1, 1, 7, 5, 1, 3, 0) (89) 
Then combining the inequalities 



(3, 


4. 


G. 


G. 


3, 


0, 


0, 


3- 


0)(20) 


dSJ 


(4, 


5. 


8. 


8. 


4, 


1, 


1, 


4. 


0)(2) 


iHD 


(3, 


3. 


1. 


1. 


7, 


5, 


1. 


3- 


0)(14) 


<EU 


(4, 


G. 


4. 


3. 


4, 


2. 


1. 


4. 


0)(40) 




(2- 


4. 


1. 


2- 


2. 


3, 


0, 


2- 


0)(5) 


d23 



and dividing by 56, we get the inequality 

(5, 7, 45/8, 5, 6, 167/56, 1, 5, 0) d45]i 

which is an improvement of (1861) . 

To derive the infinite list of inequalities given in Theorem [9] 
we will actually derive a stronger list. 

Theorem 10: For each positive integer s, the following is 
an information inequality 

(2 s - 1 - 1)1 (A; B) < 2 s - l I{A-B\C) 

+ (s -1)2 S - 2 1 (A; C\B) 

+ (s-l)2 s - 2 I(B;C\A)) 

+ (2 s - 1 - 1)I(A;B\D) 

+ ((.s-3)2 s - 2 + 1)I(A;D\B) 

+ ((s-3)2 s - 2 + 1)I{B-D\A)) 

+ (2 s - 1 - 1)I(C;D) 

Proof: Start with the Shannon inequality < I (A] B\C) 
to get the case s = 1. Once the case s has been established, 
permute the variables C, D to get 

(2 s " 1 - 1)1 (A; B) < {2 s - 1 - 1)I(A;B\C) 

+ ((s-3)2 s - 2 + 1)I{A-C\B) 

+ ((s-3)2 s - 2 + 1)I(B;C\A)) 

+ 2 S ~ 1 I(A-B\D) 

+ (s- 1)2 S - 2 1 (A; D\B) 

+ (s- 1)2 S - 2 1 (B;D\ A)) 

+ (2 s - 1 -\)I{C;D). 

Then apply Rule [6] from Section |IX] with the substitutions 



a = h = a' 


= ti 


= 2 s ' 


-!-l 


d = c = g' 


= r 


= (s 


-3)2 S ~ 2 


9 = f = c 


= d! 


= (s 


- 1)2 S - 2 




z 


= 1 




i=j = i' 


=? 


= 





to get the inequality for the case s + 1. ■ 
To see why this improves Theorem [9] note that in that 
theorem, the coefficients c\ and C2 follow the pattern: 

ci = 1, 3, 10, 34, 116, 

c 2 = 0, 1, 4, 14, 48, ... 

and both of these satisfy the recursion 

c 4 (s + 2) = 4 * Cl (s + 1) - 2 * a(s) 

and grow at the rate of 0((2 + \/2) s ) Since these majorize 
the sequences 

(s - 1)2 S ~ 2 + 2 s " 1 = 1, 3, 8, 20, 48, ... 
(s-3)2 s ~ 2 + 1 + 2"- 1 - 1 = 0, 1, 4, 12, 32, ... 

respectively, the right side of the inequality in Theorem [9] is 
strictly larger (for s > 1) than the right side of the inequality 
from Theorem [10] We note that for s = 1 both lists yield 
the Shannon inequality < I(A;B\C). For s = 2 both 
lists yield the Zhang-Yeung inequality. For s — 3 Theorem [9] 
yields Inequality d88l while Theorem [10] yields the stronger 
Inequality d43l ). 

XI. Structure of f| 

Given any four random variables, A, B, C, D we can form 
a vector of the fifteen joint entropies: 

(H(A), H(B), H(AB), H(C), H(AC), H(BC) 7 
H(ABC), H{D), H(AD), H(BD) 1 H(ABD), H(CD), 
H{ACD),H{BCD),H{ABCD)). (90) 

The space of all such fifteen dimensional vectors is called 
r^. This notation as well as some of the basic properties of this 
space are given in ifTTl . For example, this space is not closed, 
but it's closure, f 4, is convex and forms a cone with vertex 
at the origin. The spaces T\ and T| share the same interior, 
but on the boundary of these regions, T| is very complicated. 
Here we will concentrate on the closed entropy space, FJ. 

Each information inequality forms an outer bound of this 
region. The superset of f \ defined by the Shannon inequalities 
is denoted by r.4, which is also a convex cone. Note that 
although expressions such as I(A;B\C) and H(ABCD) are 
defined for probability distributions, we can easily extend 
these definitions in the natural way to any element of R 15 
using d90b . Thus, if {x\, . . . , X15) is an element of R 15 , then 
H(ABCD) = X15 while I(A;B\C) = H(AC) + H{BC) - 
H(C) - H(ABC) = x 5 +x 6 - x 4 - x 7 . 

We begin with a list of four Shannon inequalities: 

I(A;B) > 

I(A;B\C) > 

I(A;B\CD) > 

H(A\BCD) > 

These four inequalities together with the ones that can be 
formed from these by permuting the variables, form the list 
of twenty-eight elemental Shannon Inequalities. This list is 
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complete in the sense that the space they define is exactly 
(see fEQ). 

This space T4 can be reduced by combining the Shannon 
inequalities with six Ingleton inequalities, represented by the 
six permutations of: 



I(A; B) < I (A; B\C) + I(A; B\D) + I(C; D) 



(91) 



We will refer to this reduced space as the Ingleton region and 
denote it by I 4. 

The Ingleton inequalities are not information inequalities as 
defined in this paper; they are not valid for all entropy vectors. 
Their intended application, which is the study of linear ranks, 
does not concern us here. Nevertheless as we will see below, 
the Ingleton region is very important in the structure of entropy 
space. For one thing, every vertex of I4 has been shown to 
be an entropy vector for an actual probability distribution. It 
follows by convexity that I4 C Fj. Therefore, it is outside 
the Ingleton region where the non-Shannon inequalities are 
relevant. 

Since the spaces T|, T4,, and I4 are infinite cones, it is often 
convenient to study just a cross-section. ^From here on, we 
will normalize each of these spaces by using the equation 
H(ABCD) = 1. To reduce notation, we will keep the 
same names for the normalized spaces, but we will consider 
them as subsets of R 14 , since the last entry in (|90l > is held 
constant. These normalized spaces carry all the information 
of the original spaces, but are easier to describe. For example, 
whereas T4 was originally a polytopal cone in R 15 , it is now 
a poly tope in R 14 . 

Sticking out of the normalized Ingleton region are six 
"pyramid" shaped figures. There is one such pyramid for each 
Ingleton inequality. Each pyramid is defined by the points 
that satisfy all Shannon inequalities, but fail the respective 
Ingleton inequality. These pyramids are simplices in R 14 ; 
each is bounded by an Ingleton equality and fourteen Shannon 
faces. 

For our purposes, we will concentrate on just one of the 
pyramids, the pyramid associated with (|9ll , and we will 
henceforth refer to this as the pyramid. The base of this 
pyramid is formed by the Ingleton equation (I9H with "<" 
replaced by "=". In terms of the fourteen coordinates, this 
equation becomes 



Xl + X 2 + X7 + Xll + X12 = X3 + X 5 + X 6 + Xg + X W (92) 



It is also bounded by the fourteen Shannon faces: 
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There are fourteen corresponding base points of the pyra- 
mid. Each base point satisfies the Ingleton equation d92l ) and 
all but one of the fourteen Shannon equations. The base points 
are: 



1,0,1,0,1,0,1 
0,1,1,0,0,1,1 
0,0,0,1,1,1,1 
0,0,0,0,0,0,0 

111 1 

2' 2' 2' ' 2' 
1111 

' 2' 2' 2' 2' 1 ' 
1,0,1,1,1,1,1, 

0,1,1,1,1,1,1, 

1,0,1,0,1,0,1, 

0,1,1,0,0,1,1, 

1,1,1,0,1,1,1, 

1,1,1,1,1,1,1, 

112 12 2 

3'3'3'3'3'3 
1,1,1,1,1,1,1, 



0,1,0,1,0,1,0) 
0,0,1,1,0,0,1) 

0. 0.0.0.1.1.1) 

1,1,1,1,1,1,1) 
1,1,1,^1,1,1,1) 

1, ^,1,1,1,1,1) 

0,1,0,1,1,1,1) 
0,0,1,1,1,1,1) 

1,1,1,1,1,1,1) 
1,1,1,1,1,1,1) 
1,1,1,1,1,1,1) 

0,1,1,1,1,1,1) 
12 2 2 

' '3'3'3' '3' ' ' 
1,1,1,1,1,1,1). 



The remaining vertex of the pyramid will be called the 
top of the pyramid. This point satisfies all fourteen Shannon 
equations, but fails the Ingleton inequality. It is the point 

,113133 133 

(5' 2' 4' 2' 4' 4' X ' 2' 4' i' 1 ' 1 ' 1 ' 1 ^ (93) 

No point in Y4 can simultaneously fail two Ingleton inequal- 
ities and so the six pyramids have disjoint interiors. However, 
the same Shannon equation can define a face on more than 
one of these pyramids. For example, each of the six pyramids 
shares the first four Shannon faces in the list above, since 
this set is closed under permutations of the variables. Also, 
I (A; B\C) =0 has twelve forms obtained by permuting the 
variables, and each pyramid contains eight of them. However, 
the equations of the form I (A; B\CD) = and I(C; D) = 
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each have six permuted forms, and each pyramid contains 
exactly one such pair. 

The non-Shannon inequalities give us additional outer 
bounds for T|. Together, they gradually chip away each of the 
pyramids. We will often consider these inequalities in groups 
according to how many auxiliary variables are used. Thus, the 
region formed by the Shannon inequalities will be referred 
to as "0 copies". When we add the Zhang-Yeung inequality 
the corresponding outer-bound region will be referred to as 
"1 copy". Similarly, when the inequalities from Section [Vl are 
added we get the region of "2 copies", when the inequalities 
from Section El are added we get the region of "3 copies", 
and adding the inequalities from Section IVIII give us the 
region of "3.5 copies". The more inequalities we add, the 
more complicated the outer-bound region seems to become. 
Here we show how the number of inequalities (faces) and the 
number of vertices in the normalized outer bound grows with 
the number of copy steps. 



copies 


faces 


vertices 


[Shannon] 


28 


41 


1 [Zhang-Yeung] 


40 


89 


2 [Section [V) 


160 


299 


3 [Section E) 


796 


10361 


3.5 [Section EH 


4924 


224801 



One way of estimating progress toward characterizing T^ 
is by the decrease in volume of the outer bounds. Thus, the 
Shannon region, I4 , has total volume 

1.9787036156085- 10~ 10 . 

The Ingleton region has volume 

1.964365183611- 10~ 10 

leaving a volume of 

2.38973866625 • 10" 13 

for each of the six pyramids. The following table shows 
how the volume decreases within each pyramid as more non- 
Shannon inequalities are found. 



copies 


percent of pyramid left 


[Shannon] 


100 


1 [Zhang-Yeung] 


98.4568 


2 [Section |V) 


97.7040 


3 [Section El 


96.7214 


3.5 [Section NWi 


96.4682 



Of course, we do not know how much progress we are 
making without knowing the goal, which is the proportion 
of each pyramid taken up by . Although this volume is 
unknown, we can get lower bounds. All we have to do is 
find many probability distributions of four random variables, 
compute their joint entropies, obtain the corresponding ele- 
ments of R 14 and then compute the volume of the convex 
hull of these points. The problem with this technique is that 
computationally, we can only handle relatively few points. 



Nevertheless, using this technique with eight points (not 
necessarily the best eight points) we determine that the inner 
volume of f | is at least 53.4815 percent of each pyramid. It is 
clear that according to this measure, there is still quite a gap 
to fill, with room for improvement in both inner and outer 
bounds . 

Perhaps one of the most interesting entropy vectors is the 
one that yields the worst violation of Ingleton's inequality ( 19 It . 

Definition 3: Given a probability distribution, we define the 
Ingleton score of the distribution to be the value of 

I (A; B\C) + I (A; B\D) + J(C; D) - I (A; B) 
H(ABCD) 

Conjecture 1 (Four-atom conjecture): The lowest possible 
Ingleton score is approximately -.089373. It is attained by a 
four-variable distribution with alphabet size two, given by 

P(0, 0,0,0) = P(l, 1,1,1) = a 
P(0,1,0,1) = P(0, 1,1,0) = .5 -a 

where a takes on a value which is approximately 0.350457. 

Evidence for this conjecture comes from hill-climbs. Using 
Newton's method, we climb from random starting distributions 
using alphabet sizes from 2 to 10. We gradually make changes 
to the distribution in order to decrease the Ingleton score. 
The most successful climbs, i.e. the climbs that result in the 
lowest score, always tend to the four-atom distribution given 
in the conjecture, some permutation of it, or a direct product 
of independent copies of such distributions. In other words, 
we easily find the score —.089373 over and over again, but 
never beat it. 

This suggests another way of measuring progress by how 
close we come to proving the Four-Atom conjecture. Given 
any outer-bound region, we can consider the lowest Ingleton 
score allowed in the region. Progress toward the Four-Atom 
Conjecture is given in the following table: 



copies 


minimum Ingleton score 


[Shannon] 
1 [Zhang-Yeung] 

2 [Section [V) 

3 [Section El 
3.5 [Section ITU 


-1/4 « -.25000 
-l/6« -.16667 
-l/6« -.16667 
-7/44 « -.15909 
-3/19 « -.15789 



Again, progress toward the conjectured goal of -.089373 
seems slow. In fact, even if we combine all the non-Shannon 
inequalities in this paper, it is still not enough to prove the 
four-atom conjecture and it is likely that some new techniques 
will be necessary to settle it. 

We now present a third way of measuring progress of the 
non-Shannon inequalities. Unlike the previous two methods, 
this method makes explicit use of the infinite lists of inequal- 
ities from Section |IX] We begin with some definitions. 

Definition 4: Let P be a poly tope in R n , let L be an affine 
function of n-variables such that for each point x in P, L(x) > 
0. Then the set of points in P satisfying L(x) = is called an 
extreme segment of P. Thus, the vertices of P are the extreme 
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points of P, the edges of P are the extreme line segments of 
P, etc. 

Definition 5: Let P be a polytope in R™ and let < m < 
n. Then the m-skeleton of P is the union of all of the extreme 
TO-dimensional segments of P. 

Definition 6: Let P be a polytope in R n , let < m < n, 
and let S be a subset of P. Then the m-core of S in P is the 
convex hull of the intersection of S with the m-skeleton of P. 

It is easy to see from the Zhang-Yeung inequality that I4 is 
the 0-core of T| in T4. First of all, the polytope I4 is a subset 
of both T| and T^, so we only need to show that TJ does not 
contain any vertices of T4 that lie outside of I 4. Outside of 
I4 the only vertices of T4 are the tops of the pyramids. But 
these tops fail one of the permuted forms of the Zhang-Yeung 
inequality. For example, the top of the pyramid give by (l93t 
fails the inequality of Theorem [3] 

We will now use the Zhang-Yeung inequality to show that 
I4 is also the 1-core of F4 in T4, Then we will use the 
infinite families of inequalities to prove the stronger result 
that it is also the 3-core. So the Ingleton region, aside from its 
applications to linear ranks, is a natural part of the geometry 
of entropy space. 

Theorem 11: The 1-core of T\ in T4 is I 4. 
Proof: Since the polytope I4 is a subset of both T| and 
T4, we need to show that the intersection of with the 1- 
skeleton of T4 lies entirely in I 4. 

We only need the Zhang-Yeung inequality, 

I(A;B) < 2I{A-B\C)+I{A;C\B)+I(B-C\A) 
+ I(A;B\D)+I{C;D) 

and the Ingleton inequality (19Tb . 

I(A; B) < I(A; B\C) + I(A; B\D) + I(C; D) . (94) 
Subtracting the two, we get the inequality 

< I(A; B\C) + I(A; C\B) + I(B; C\A) 

which must be satisfied by any point on the base of the 
pyramid, since the Ingleton equation (l92l l is satisfied by such 
points. Also, if any base point has a zero value for all of the 
terms 

I(A;B\C),I(A;C\B),I(B;C\A) 

Then it satisfies the Zhang-Yeung inequality non-strictly (as 
an equation). 

We now consider an extreme line segment connecting a two 
vertices of T4, one of them being the top of the pyramid. Since 
this is the only vertex of T4 that fails the Ingleton inequality, 
the other vertex satisfies the Ingleton inequality. Therefore, at 
some point on this segment, the Ingleton equation is satisfied. 
We will call this the base point of the segment. 

There are thirteen Shannon equations satisfied by all of 
the points on this segment. Since the top of the pyramid is 
defined by the fourteen Shannon faces of the pyramid, the 
thirteen Shannon equations satisfied by the line segment must 
be included in this group of fourteen. Therefore, all points 
on the line segment are either zero on all three of the terms 
above, or on all three of the terms with C, D interchanged, 

I(A;B\D),I(A;D\B),I(B;D\A), 



since these represent six of the fourteen Shannon faces. 
Therefore, the base point of this line segment either satisfies 
the Zhang-Yeung equation or the Zhang-Yeung equation with 
C, D interchanged. On the other hand, it is easy to check 
that the top of the pyramid d93l fails both the Zhang-Yeung 
inequality, and the same inequality with C, D interchanged. 
By linearity, one of the two Zhang-Yeung inequalities must 
fail everywhere on the part of the line segment where the 
Ingleton inequality fails. Therefore the intersection of F| with 
the 0-skeleton of T4 lies entirely in I 4. ■ 

We will now prove a stronger result using a similar tech- 
nique, and making use of the infinite families of inequalities. 

Theorem 12: The 3-core of f 4 in T4 is I4. 

Proof: As in the previous proof, we need to show that no 
point of r|\ I4 is in the 3-skeleton of T4. We suppose there is 
such a point and call it x. Then x lies on a three dimensional 
extreme segment that includes at least four vertices of T4, one 
of which we may assume is the top of the pyramid and is 
the only one that fails the Ingleton inequality. The segment 
is defined by eleven Shannon faces, and since the segment 
includes the top of the pyramid, these eleven faces must be 
among the fourteen Shannon faces of the pyramid. 

Suppose first that the eleven faces includes the two faces 

I{A-B\C) = 
I(A;C\B) = 0. 

Then consider the first Matus family of inequalities ( |80t 

sI(A;B) <= (s(s + 3)/2)I(A;B\C) 

+ (s(s + l)/2)I(A;C\B)+I(B;C\A) 
+ sI(A;B\D)+sI(C;D), 

which on our segment reduces to 

<= -I(A;B) + (l/s)I(B;C\A) 
+ I(A;B\D)+I(C;D). 

At the top of the pyramid, all terms of this reduced 
inequality are zero except I(A; B). The inequality becomes 
< —1/4, which is false. But if the Ingleton equation is 
satisfied, then this becomes 

<= (l/s)I(B;C\A). 

and the value of the right side tends to zero with large values 
of s. By linearity, the value of the right side of the reduced 
inequality at x must become negative for sufficiently large s, 
so the information inequality is false. Therefore, the eleven 
defining faces of the segment cannot include both I (A; B\C) 
and I(A;C\B). 

A similar argument with A and B interchanged tells us 
that the eleven faces cannot include both I(A;B\C) and 
I(B;C\A). 

Next, suppose that the eleven faces of our segment includes 
the two faces 

I{A-C\B) = 
I(B;C\A) = 0. 
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Iterating Rule 2, and starting with the Shannon Inequality < 
I(A;B\C), we generate the infinite list of inequalities: 

sI(A;B) < {s + l)I(A;B\C) + (s(s + l)/2)I(A;C\B)) 
+ (s(s + 1)/2)J(B; C\A)) + sI(A; B\D) 
+ sI(C; D) , 

where s is any any non-negative integer. On our segment this 
reduces to 

< -I(A;B) + (s + l)/sI(A;B\C)+I(A;B\D) 
+ I(C;D). 

At the top of the pyramid, this again becomes < —1/4, 
which is false. If the Ingleton equation is satisfied, then this 
becomes 

< (l/s)I(A;B\C), 

We conclude, as before, that the inequality becomes false 
at x for sufficiently large values of s. Therefore, the eleven 
defining faces of our segment cannot include both I(A; C\B) 
and I(B;C\A). 

Summarizing the work so far, we see that the eleven faces 
cannot include any two of the three faces 

I(A;B\C) = 
I(A;C\B) = 
I(B; C\A) = 0. 

A similar argument with C and D interchanged shows that 
the eleven faces cannot include any two of the three faces 

I(A;B\D) = 
I{A;D\B) = 
I{B-D\A) = 0. 

But only three of the fourteen Shannon faces are excluded, 
so this is a contradiction. ■ 

Future inequalities may be able to improve this further, 
perhaps showing that I4 is the 4-core of r| in T4, etc. But it 
can't go beyond the 5-core since the four-atom point lies on 
eight of the fourteen Shannon faces. 

XII. Application of Non-Shannon Information 
Inequalities to Network Coding 

One application of non-Shannon information inequalities is 
in the field of network coding. In network coding, a subset of 
network nodes called sources generate messages and a subset 
of network nodes called receivers need to acquire the source 
messages. The messages can propagate through the network 
but need not travel as in packet-switched routing. Rather, 
network "coding" allows every out-edge of a node to be an 
arbitrary function of the information carried on the node's in- 
edges. 

In this section, a network is a finite, directed, acyclic 
multigraph together with a finite set called the message set. 
An alphabet is a finite set A with at least two elements. Two 
special subsets of nodes of the network are called sources and 
receiver, respectively. The sources generate messages and the 



receivers need to obtain certain source messages (namely those 
messages that the receiver demands). 

Let fc and n be positive integers, called the source dimension 
and the edge capacity, respectively. For every network edge 
e = (x, y) an edge function f e puts an n-dimensional vector 
of alphabet symbols on e; the vector's value is a function of 
the n-dimensional vectors carried on the in-edges of node x 
and the fc-dimensional source vectors produced at x. Similarly, 
for every receiver node x and every network message m 
demanded by x, a decoding function f x ,m produces a fc- 
dimensional vector as a function of the in-edge and source 
values at x. 

Given an alphabet A, a (k,n) code for a network is an 
assignment of edge functions and decoding functions to the 
network's edges and receivers, respectively. 

A (fc, n) network code is said to be a (fc, n) solution if every 
receiver can recover each of the messages it demands (i.e. via 
it's decoding functions). 

Special codes of interest include linear codes, where the 
edge functions and decoding functions are linear, and routing 
codes, where the edge functions and decoding functions simply 
copy input components to output components. 

If a network has a (fc, n) solution over some alphabet, then 
we say the ratio k/n is an achievable coding rate for the 
network. 

The coding capacity of a network is 

sup{fc/n : 3 (fc, n) coding solution over ^4}. 

If (fc, n) coding solutions are restricted to linear codes (over 
some finite field, for example) or routing codes, then the 
capacity is called the linear capacity or routing capacity, 
respectively. 

Computing the network coding capacity of networks, or 
at least accurately approximating or bounding the capacity 
is a fundamental problem in network coding. No general 
method for computing network capacity is presently known, 
so attention is focused on bounding techniques. Lower bounds 
are generally obtained by exhibiting specific solutions for 
a network. Upper bounds can sometimes be obtained by 
assuming the source messages are random variables and then 
using standard information inequalities. 

Specifically, in Q, the Zhang- Yeung inequality (our The- 
orem Q3 was used to derive an upper bound on the coding 
capacity of the Vamos network (see Figure [T}. The Vamos 
network is a network constructed from the well-known Vamos 
matroid and is of interest as it is presently the only known 
network for which non-Shannon information inequalities have 
improved capacity bound calculations. 

The capacity upper bound derived in ]3| was 10/11 and 
was shown to be strictly smaller than the smallest possible 
upper bound (i.e. 1) obtainable directly from Shannon-type 
information inequalities. The Vamos network thus illustrates 
a potential for improvement in capacity calculations using non- 
Shannon-type inequalities over Shannon-type ineqaulities. 

It was also shown in Q that the linear coding capacity of 
the Vamos network over every finite field is exactly equal to 
5/6, which is presently the best known lower bound on the 
(possibly non-linear) coding capacity of the Vamos network. 
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The capacity bound calculation in ||3) was specific to 
the particular form of the Zhang-Yeung inequality. A more 
general strategy for improving capacity upper bounds using 
other non-Shannon inequalities is given in what follows. We 
present a systematic method for using non-Shannon informa- 
tion inequalities to obtain capacity bounds. In particular, the 
best known upper bound on the Vamos network capacity is 
improved from 10/11 to 19/21. The exact coding capacity of 
this network remains an open problem, lying somewhere in 
the interval [5/6, 19/21]. 
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Fig. 1 . The Vamos network. A message variable a, b, c, or d labeled above 
a node indicates an in-edge (not shown) from the source node (not shown) 
generating the message. Demand variables are labeled below the receivers 
n-9— ni3 demanding them. The edges ei : 2> 63,4, £5,6. an d e 7,8 are denoted 
by w, x, y, and z, respectively. 

We will now demonstrate one way to exploit the new non- 
Shannon inequalities given in the present paper to obtain even 
tighter bounds on the coding capacity of the Vamos network. 
The next theorem provides a tighter upper bound on the Vamos 
network coding capacity than previously known. 

Theorem 13: The coding capacity of the Vamos network is 
at most 19/21. 

Proof: Consider a (k, n) solution to the Vamos network. 
Assume that the network messages a, b, c, d are independent, 
/c-dimensional, random vectors with uniformly distributed 
components and assume each edge in the network has capacity 
n. Let w, x, y, z denote the random variables carried by edges 
ei,2j £3,4) e 5.6, e 7.8, respectively. Since the n-dimensional vec- 



tor carried on any edge e = (it, v) is a deterministic function 
of the in-edges to node u and sources generated at u, the 
conditional entropy of the random vector on e, given the 
vectors on the in-edges to u and message vectors produced 
at u must be zero. Thus, we have 



H(w\a 7 b,c,d) = 
H(x\a,b,w) = 
H(y\b,c,x) = 
H{z\c, d, y) = 0. 



[from n i] 
[from 713] 
[from 715] 
[from 717] 



(95) 
(96) 
(97) 
(98) 



Similarly, since the fc-dimensional vector decoded at a receiver 
u is a deterministic function of the in-edges to u and sources 
generated at u, the conditional entropy of the decoded vector at 
u, given the vectors on the in-edges to u and message vectors 
produced at u must be zero. Thus, we have 



H(b\d,x,z) = 
H{a\b,c, d,z)=0 
H(b, c\a, d, w, z) = 
H(d\a,b, c,y) = 
H(c\a,w,y) = 0. 



[from Tig] 
[from mo] 
[from nn] 
[from 7112] 
[from m 3 ] 



(99) 
(100) 
(101) 
(102) 
(103) 



In what follows we repeatedly make use of the following 
well known facts: 



I(R; S\T) = H(R, T) + H{S, T) 
- H(R,S, T) - H{T) 
H(R,S) = H(R) + H(S\R) 
H{R,S) < H(R) + H{S) 
H{R) < H{R,S). 

We thus obtain the following list of inequalities: 

I(c,y;b,x) 

= H(c, y) + H(b, x) - H{b, c, x, y) [from dl04t ] 
= H(c, y) + H(b, x) - H(b, c, x) [from @7j] 



(104) 
(105) 
(106) 
(107) 



>H(c,y)-H(c) 

I(c,y;b,x\d,z) 

= H(c, d, y, z) + H{b, d, x, z) 

— H{b, c, d, x, y, z) — H(d, z) 
= H{c, d, y) + H(d, x, z) 

— H(a, b, c, d, x, y, z) — H(d, z) 
< H{c,y) + H(d)+H(x) 

— H(a, b, c, d) 

I(c,y;d,z\b,x) 

= H(b, c, x, y) + H{b, d, x, z) 

— H(b, c, d, x, y, z) — H(b, x) 
= H(b, c,x) +H(d,x,z) 

— H(a, b, c, d, x, y, z) — H(b, x) 
< H(c) + H(d,z) + H(x) 

— H(a, b, c, d) 



[from COS] (108) 



[from CjjjS ] 

[from (|99),(|98)] 

[from (fTO0li.(l95ll.(l96ll] 

[from (fT06l >] 

[from C07l»] (109) 



[from (TT04b ] 
[from (|97},([99}] 
[from ([TOOt ] 
[from ([T06l l] 
[from (d07]i] (110) 
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I(b,x;d,z\c,y) 

= H(b,c,x,y) + H(c, d,y,z) 

- H(b,c,d 7 x,y,z) - H{c,y) [from JT041] 

= H(b,c,x) + H{c,d,y) [from g3,@8)] 

- H(a,b,c,d,x,y, z) - H(c,y) [from ( 1 100b ] 

< ff(6, x) + ff(c) + (d) [from (fT06l i] 
-H(a,b,c,d) [from (TTOTll (111) 

Z(c, y; 6, »|o, to) 

= -ff (a, c, to, y) + -ff (a, b, w, x) 

— H(a,b,c,w,x,y) — H(a,w) [from ( fT04l ] 

= H(a, w, y) + H(a, b, w) [from (fT03l.(l96ll] 

— H(a,b,c,d,w,x,y) — H(a,w) [from < fT02T >] 

< H(y) + H{a, w) + H(b) [from lfl06t ] 
-H(a,b,c,d) [fromQ23] (112) 

I(c,y;a,w\b,x) 

= H(b, c, x, y) + iZ(a, w, b, x) 

- H(a,w,b,c,x,y) — H(b,x) [from dl04b ] 

= H(b,c,x) + H(a,b,w) [from®,®] 

— H(a,b 7 c,d,w,x,y) — H(b,x) [from ( [T02l ] 

< £T(c) + If (a, to) + Jf(6) [from fl06l ] 
-H(a,b,c,d) [fromflOTJ] (113) 

/(o,x;a,to|c, y) 

= F(6, c, x, y) + ff(a, c, w, y) 

- H(a,b,c,w,x,y) - H(c,y) [from (1 104b ] 

= (&, c, x) + ff(a, io, y) [from d97li.(fT03l] 

- H(a,b,c,d,w,x,y) - H(c,y) [from (1 100b ] 
< H{b,x) + H(c) + H{a,w) + H{y) [from (H®] 

-H(a,b,c,d)-H(c,y) [from (Q23] 

(114) 

7(d, z; a, to) 

= (d, z) + ff(a, to) - ff(a, d, to, 2) [from dT04l ] 
= H(d,z) +H(a,w) 

- H(a,b,c,d,w, z) [from ( IToTb ] 

= ff(d, z) + ff(a, to) - (a, 6, c, d) [from (|95)-(|98)] 

(115) 

7(d, z; a, to|c, y) 

= #(c, d, y, z) + if (a, c, to, y) 

- H(a,c,d,w,y,z) — H(c,y) [from ( 1 104b ] 

= H(c,d,y)+H(a,w,y) [from <t98t.<fT03T>] 

— H(a,b,c,d,w,y, z) — H(c,y) [from ( fT0TT >] 

< #(d) + #(a, to) + H(y) [from < fT06t ] 
-H(a,b,c,d) [from 023] (116) 



J(d, z; a, to|&, x) 

= -^(6, d, x, z) + H(a, 6, to, x) 

- H(a,b,d,w,x, z) - H(b,x) [from (1104b ] 

= -ff (d, x, z) + -ff (a, b, w) [from ([99j,(|96j] 

- H(a, b, c, d, 10, x, z) — 7J (6, x) [from ( 1 10 lb ] 
< iJ(d, z)+H(x)+H(a, w)+H{b) [from ( fT06b ] 

- H(a,b,c,d)- H(b,x). [from 023] (117) 

Now, suppose that A, i?, C, and D are random variables 
and we have an information inequality of the form 

ai I(A; B) 

< a 2 I(A; B\C) + a 3 I(A; C\B) + a 4 I(B; C\A) 
+ a 5 I(A; B\D) + a 6 I{A; D\B) + a 7 I(B; D\A) 
+ a 8 I(C;D)+a 9 I(C;D\A)+a 1Q I(C;D\B) (118) 

for some a, > 0, for all i. If we set 

A = (c,y) 
B = (b,x) 
C = (d,z) 
D = (a, to) 

then (I108b - (ll 17b give the inequality 

ai (H(c,y)-H(c)) 

< a 2 (H(c, y) + H{d) + H(x) - H(a, 6, c, d)) 
+ a 3 (H(c) + H(d, z) + H(x) - H(a, b, c, d)) 
+ a 4 (H(b, x) + H(c) + H(d) - H(a, b, c, d)) 
+ a 5 (H(y) + H(a, w) + H{b) - H(a, b, c, d)) 
+ a 6 (-ff(c) + H{a, w) + H(b) - H(a, b, c, d)) 
+ a 7 (-H(a, b, c, d) - F(c, y)) 
+ a 8 (H(d 7 z) + H(a, w) - H(a, b 7 c, d)) 
+ a 9 (F(d) + ff(a, to) + iJ(y) - H(a, b, c, d)) 
+ aioCff (6) - ^( a > 6, c > d ) - »)) 

or, equivalently, after collecting terms, 

(a 2 + a 3 + a A )H(a) 

+ («2 + as + a$ + a 9 + a w )H(b) 

+ (05 + a 7 + a 8 + a 9 + oio)-H'(c) 

+ (a 5 + a 6 + aj)H(d) 

+ (a 2 -ai - a 7 )/(c; y) 

+ (a 4 + a 7 — ai )/(6; x) 
< (a 5 + a 6 + a 7 + a 8 + a 9 + a 10 )H(w) 

+ (a 2 + a 3 + a 4 + a 7 )iJ(x) 

+ (— ai + a 2 + a 5 + a 9 )H(y) 

+ (a 3 +a 8 + a 1Q )H(z). (119) 

If the inequalities 

a 2 > 01 + a 7 

a 4 + a 7 > aio (120) 
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are satisfied, then the inequality (11 19b directly leads to a 
Vamos coding capacity region bound, by neglecting the (non- 
negative) terms involving I(c; y) and I(b; x). Specifically, in 
this case, by substituting 

H(a) = H(b) = H{c) = H(d) = k 

H(w) = H(x) = H(y) = H{z) = n 

into ( 11191 ), we obtain 

fc(2a 2 + 2a 3 + a 4 + 2a 5 + a 6 

+ 2a 7 + 2a 8 + 2a 9 + 2a w ) 

< n(—ai + 2a2 + 2a 3 + 04 + 2as 

+ a 6 + 2a 7 + 2a 8 + 2a 9 + 2ai ). (121) 

One of our new non-Shannon inequalities is 

2I(A; B) < 3I(A; B\C) + 2I(A; C\B) 

+ 3/(5; C\A) + 2I{A; B\D) + 2I(C; D) (122) 

which is equivalent to inequality ( f52b (by exchanging the roles 
of A and B in in Theorem |6). Using (II 18b and (11221 1. we see 
that both of the inequalities in ( 1120b hold and therefore we 
substitute 



(ii) 



a L 


= a 3 


= C15 


= a s 


«2 


= (14 


= 3 




a e 


= a-7 


= a g 


= aw 



into ( 11211 ) and obtain 



k/n < 19/21. 



(123) 



Thus, the coding capacity of the Vamos network can be at 
most 19/21. ■ 

We note that if one or both of the inequalities in (1120b 
are not satisfied, we may be able to combine dl 19b with 
another such inequality (having a positive coefficient for 
I(c;y) and/or I(b;x)) to eliminate the extra term(s). Some 
specific inequalities that are useful for this are given in the 
following lemma. 

Lemma 14: 



ii) 


H(a)- 


hI(c;y)<H(y) 


(ii) 


H{a)- 


1- 1(6; x) < H{x) 


(Hi) 


H(a)- 


VH(b)+I(c-y)<H(x)+H{y) 


(iv) 


H(b)-\ 


-H(c) < H(w) + H(y) 


(v) 


H(c)-\ 


-H{d) < H(w)+H(y) 


(vi) 


H(d)- 


h I(c;y) < H(y) 


(vii) 


H(d)- 


\-I(b;x) < H(x) 


Proof: 







(i) 



H(y)-H(a)-I(c;y) 
= H(c,y)-H(a)-H(c) 

> H(b, c, d, y) — H(a, b, c, d) 

> H(b, c, d, z) — H(a, b, c, d) 

> H(a, b, c, d) — H(a, b, c, d) 
= 



[from (U04b 
[from (IIOSV 
[from (IS] 
[from (HOOT 



(iii) 



(iv) 



(v) 



(vi) 



(vii) 



H(x) - H(a) - I(b; x) 
= H(b,x) - H(a) - H(b) 

> H(b, c, d, x) — H(a, b, c, d) 

> H (6, c, d, y) — H(a, b, c, d) 

> 



H(x) + H(y) - H(a) - H(b) 
-I(c;y) 
= H(c, y)+H(d) + H(x) 
— H(a, b, c, d) 

> H(c, d, x, y) — H(a, b, c, d) 

> H(c, d, x, z) — H(a, b, c, d) 

> H(a, b, c, d) — H(a, b, c, d) 
= 



[from COD] 
[from dlOSll ] 
[from (|97j] 
[from (flM ] 



[from ( fT04b ] 
[from (fT06l >] 
[from (|98])] 
[from (t99l.(fT00b] 



H(w) - H(y) - H(b) - H{c) 

> H(a, d, w, y) — H(a, b, c, d) 

> H(a, d, w, z) — H(a, b, c, d) 

> H(a, b, c, d) — H(a, b, c, d) 
= 



H(w) - H(y) - H(c) - H(d) 

> H (a, b, w, y) — H(a, b, c, 0?) 

> i?(a,6, c, d) - i/(a,6,c, d) 
= 



[from (fT06b ] 
[from (fT03l.(f98ll] 
[from (TToTb ] 



[from (fT06t ] 
[from (fT03l.(fT02l] 



H(y)-H(d)-I(c;y) 
= H(c,y)-H(c)-H(d) 

> H (a, b, c, y) — H(a, b, c, d) 

> H(a, b, c, d) — H(a, b, c, d) 
= 



H(x) - H{d) - 7(6; x) 
= H{b,x) - H(b) - H(d) 

> H (a, 6, c, a;) — -ff (a, 6, c, d) 

> (a, 6, c, y) — (a, 6, c, d) 

> 0. 



[from ( fT04b ] 
[from (fT06T >] 
[from dT02l ] 

(125) 



[from (fT04b ] 
[from (fT06j] 
[from (f97])] 
[from (fT25t ] 



(124) 



We have exhaustively computed network coding capacity 
bounds for the Vamos network using all of the new non- 
Shannon information inequalities we have discovered and the 
bound in (1 123b is the tightest (i.e. smallest) among those we 
checked. 
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XIII. Conclusion 

In this article, we have presented a lengthy list of non- 
Shannon type inequalities. This is a continuation of work 
started by Zhang and Yeung and roughly follows their ap- 
proach. We have seen that many of the inequalities, including 
the original Zhang-Yeung inequality have been superseded by 
stronger ones. Almost certainly, many more of the inequalities 
that have been presented here will also be superseded by 
future work in this area. To our knowledge, no one has ever 
given a non-Shannon inequality and proved that it will not be 
superseded by others. This would certainly be an interesting 
challenge for future research. 

Although each inequality found gives more information 
about the shape of entropy space, and each one is at least 
theoretically useful in a network information flow problem, it 
seems unlikely that endlessly producing new inequalities will 
be fruitful in itself. These inequalities are presented with the 
hope that it may be possible to study the list as a whole, in 
order to gain further insights that will enable doing more than 
just endlessly extending the list. 

Along these lines, perhaps the most striking feature of list 
of the inequalities listed is their special form. Indeed every 
inequality produced has the form 

aI(A;B) < bI(A; B\C) + d(A;C\B) + dI(B;C\A) 
+ eI(A;B\D) + fI(A;D\B)+gI(B;D\A) 
+ hI(C;D) + U(C;D\A) 

and vary from each other only by the choice of the coefficients 
a, b, c, d, e, /, g, h, i. There seems to be no obvious reason for 
this form. Indeed, there were a few inequalities that were 
found and were not of this form. But each of these non- 
conforming inequalities was superseded by other inequalities 
or was reduced to the special form under tightening. Explain- 
ing this special form might be a first step in gaining a deeper 
understanding of the inequalities. 

Another direction to pursue is to find patterns among the 
inequalities and/or their proofs. Identifying such patterns may 
allow us to continue the patterns indefinitely. Along these 
lines, we have identified several "rules" that allow us to 
generate new inequalities from old ones. Application of these 
rules quickly yields infinite families of inequalities similar to 
those found by Matus |__. Families of inequalities have already 
given valuable insight into the structure of entropy space and 
were instrumental in Matus' proof that entropy space is not 
a polytope (or polytopal cone). To our knowledge, no one 
has yet been able to show whether or not entropy space is 
curved. This would certainly be a valuable contribution toward 
understanding this space. 

Finally, the techniques used in this paper seem limited and 
do not seem like they will be strong enough to settle the 
four atom conjecture. We believe therefore, that some new 
techniques for finding non-Shannon inequalities will be vital 
for gaining a more complete understanding of entropy space. 
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